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Executive  Summary 


The  goal  of  this  research  effort  was  to  develop  quantitative  and  scientific  methods  that 
capable  of  assisting  researchers  and  military  logisticians  in  the  analysis  of  the  effectiveness  of 
logistics  technology  and  processes  in  a  Sense  &  Respond  Logistics  (S&RL)  environment  that 
incorporated  both  a  human-centered  and  a  system-based  perspective.  Because  of  the  complexities 
involved  with  S&RL  concepts,  it  is  imperative  that  scientific  and  engineering  approaches  based 
on  a  holistic  and  systems  engineering  perspective  be  utilized  to  improve  logistics  support  and 
responsiveness  within  this  evolving  defense  logistics  archetype.  The  logistics  support  function 
should  be  treated  as  a  socio-technical  system  where  human  elements,  technology,  and 
software/algorithms  interact  to  develop  solutions  that  can  impact  the  entire  system.  To  this  end, 
with  the  assistance  of  the  Logistics  Readiness  Branch  of  the  Air  Force  Research  Lab 
(AFRL/HEAL),  Air  Mobility  Command  (AMC)  and  the  United  States  Transportation  Command 
(USTRANSCOM),  a  human-centered  research  initiative  consisting  of  eight  distinct  research 
efforts  designed  to  investigate  and  support  the  tenants  of  a  S&RL  paradigm  for  the  United  States 
Air  Force  was  developed.  The  specific  research  efforts  included:  1)  Effective  Scheduling  and 
Coordination  of  Disaster  Relief  Operations,  2)  Experimental  Evidence  on  Team  Coordination 
and  Collaboration  Within  a  Distributed  Logistics  Network,  3)  A  New  Paradigm  For  Studying 
Trust  In  Virtual  Teams,  4)  Developing  and  Evaluating  Operationally  Robust  Forecasting 
Techniques  in  Military  Logistics,  5)  Integrated  Distribution  Planning  and  Forecasting  for 
Medical  Logistics,  6)  An  Auction-Based  Framework  for  Resource  Allocation  in  Disaster  Relief 
Operations,  7)  Comparing  Maintenance  Policies  for  Single-Unit,  Markovian  Systems,  and  8) 
Improved  Outbreak  Detection  for  Bio  Terror  Response  Logistics.  Key  contributions  from  this 
effort  include:  1)  New  modeling  paradigms  for  supply  chain  modeling  and  analysis  in  support  of 
disasters,  2)  New  simulation  methodologies,  mathematical  models  and  techniques  for  the 
optimization,  perfonnance  evaluation,  and  improvement  of  future  military  logistics  support 
networks,  and  3)  New  methods  of  modeling  and  incorporating  human  performance  issues  and 
collaborative  decision  making  within  an  S&RL  environment.  During  the  period  of  performance 
for  this  research  effort,  a  total  of  15  graduate  research  assistants,  4  undergraduate  research 
assistants,  and  1 1  faculty  members  were  supported.  The  research  efforts  have  resulted  in  1 
related  thesis,  7  related  dissertations,  14  conference  papers,  and  1 1  journal  submissions. 
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1 . 1  Executive  Summary 

The  United  States  (US)  Transportation  Command  (TRANSCOM)  is  the  US  Department 
of  Defense’s  primary  distribution  agent  for  transporting  military  troops  during  deployment 
activities  and  delivering  disaster  relief  supplies  during  humanitarian  aid  operations. 
Transportation  and  scheduling  decisions  are  vital  for  supplying  personnel,  food,  medical 
supplies,  and  building  materials  to  an  area  affected  by  a  disaster — they  impact  how  and  when 
these  critical  items  arrive  in  theater.  Moreover,  the  capabilities  and  associated  capacities  of 
available  transportation  vehicles  (resources)  add  an  additional  layer  of  complexity  into  the 
transportation  scheduling  problem.  In  an  attempt  to  minimize  transportation  system 
inefficiencies,  we  present  an  optimization  model  for  scheduling  transportation  operations  that 
incorporates  the  important  practical  considerations  of  dependency  and  precedence  relationships 
between  items.  Our  model  seeks  to  minimize  inventory  holding.  Initial  experimental  results 
suggest  the  proposed  optimization  model  is  an  attractive  alternative  to  current  simulation-based 
theater  distribution  models. 
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1 .2  Introduction 


Troop  deployment  and  humanitarian  aid  operations  in  air  mobility  are  a  series  of  efforts 
conducted  by  interconnected  transportation  and  logistics  systems  composed  of  airlift,  air  fueling, 
and  air  mobility  support  (Hazdra,  2001).  Airlift,  which  is  the  focus  of  this  research,  consists  of 
the  transportation  of  military  and  civilian  personnel  (passengers),  materials  (cargo),  and  supplies 
(cargo)  through  air  when  and  from/to  where  they  are  needed.  In  a  disaster  relief/humanitarian  aid 
environment,  passengers  may  be  doctors,  construction  personnel,  or  relief  workers,  while  cargo 
may  consist  of  medical  supplies,  construction  equipment,  or  food.  These  passengers  and  cargo 
are  transported  from  their  initial  staging  locations  (e.g.,  military  bases,  hospitals,  etc.)  to  the 
impacted  geographic  area(s)). 

The  United  States  Transportation  Command  (USTRANSCOM)  is  responsible  for  the 
transportation  and  delivery  of  passengers  and  cargo  for  the  US  Department  of  Defense.  In  a 
typical  transportation  scenario,  USTRANSCOM  receives  requirements  and  infonnation  on  the 
passengers  and  cargo  to  move,  as  well  as  origin  and  destination  information.  Scheduling  data 
pertaining  to  1)  when  items  to  be  transported  are  available  at  the  origin  and  2)  when  these  same 
items  are  due  at  the  destination  are  also  essential  pieces  of  information  for  USTRANSCOM. 
Considering  all  available  data,  USTRANSCOM  plans  and  executes  item  transportation  and 
distribution. 

Transported  items  often  are  of  variable  size,  dimension,  or  priority/importance,  and  have 
dissimilar  planned/required  arrival  dates  on  location.  Some  key  issues  that  must  be  considered 
for  successful  delivery  is  item  compatibility  between  items  being  transported.  For  example,  some 
pairs  of  items  are  not  compatible  to  occupy  the  same  aircraft  at  the  same  time  (e.g.,  medical 
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equipment  should  not  be  on  the  same  aircraft  as  ammunition  due  to  potential  contamination  of 
the  medical  equipment). 

These  considerations  are  essential  for  rapid,  efficient  disaster  relief  operations  and  should 
be  considered  simultaneously  for  maximum  efficacy.  Disaster  relief  operations  efficacy  is  a 
function  of  both  item-to-transportation  vehicle  (e.g.,  aircraft)  assignment  and  vehicle  scheduling 
decisions.  Considering  that  scheduling  transportation  resources  in  disaster  relief  operations  is 
already  a  logistics  challenge,  focused  research  is  needed  into  integrated  cargo  load 
building/formation  (i.e.,  multiple  items  of  different  attributes  are  grouped  to  “form”  loaded 
aircraft)  and  scheduling  (cargo  transportation  and  distribution  assets  are  scheduled  in  complex 
logistics  networks)  problems. 

In  this  study,  we  investigate  the  use  of  a  time-indexed  mathematical  model  to  analyze  this 
important  and  integrated  problem.  While  our  primary  motivation  comes  from  disaster  relief 
operations  scheduling,  it  is  important  to  note  that  another  application  of  this  same  integrated  load 
building  and  scheduling  problem  is  troop  deployment.  When  battalions/companies/divisions  of 
soldiers  are  deployed  to  a  combat  theater  or  other  location,  their  associated  equipment,  gear,  and 
weapons  systems  must  be  transported  effectively  and  efficiently  using  USTRANSCOM 
resources.  In  this  case,  various  item  dependence  and  precedence  relationships  can  also  exist. 

Our  proposed  model  provides  decision  makers  with  the  ability  to  deliver  timely  for  any 
desired  number  of  available  transportation  resources.  The  remaining  sections  of  this  paper  are 
organized  as  follows.  Section  2  begins  by  defining  key  tenns  in  air  mobility  operations,  and  then 
reviews  the  current  literature  pertaining  to  military  applications.  Our  proposed  mathematical 
programming  model  and  some  extensions  are  presented  in  Section  3.  Section  4  explains  the 
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heuristic  development,  and  Section  5  discusses  our  experimental  design  and  results.  Finally, 
research  conclusions  and  directions  for  future  research  are  provided  in  Section  6. 

1 .3  Literature  Review 

There  exists  a  considerable  amount  of  previous  research  related  to  item-to-vehicle 
assignments  and  mission  scheduling  for  different  military  operations.  A  number  of  researchers 
try  to  capture  the  dynamic  aspects  of  deployment  operations  using  optimization-based 
techniques.  Two  primary  classes  of  optimization-based  techniques  are  evident:  mathematical 
optimization  and  heuristic  optimization. 

1.3.1  Previous  Mathematical  Programming-Based  Optimization  Research 

Morton  et  al.  (1996)  present  a  multi-period,  multi-commodity,  network-based 
optimization  formulation  containing  demand  satisfaction,  aircraft  balance,  aircraft  capacity, 
aircraft  utilization,  and  airfield  capacity  constraints.  The  model’s  objective  is  to  minimize  total 
weighted  penalties  for  late  deliveries  and  non-deliveries.  If  sufficient  resources  exist  to  meet  all 
delivery  targets,  then  the  model  seeks  to  minimize  the  number  of  aircraft  used.  Rosenthal  et  al. 
(1997)  apply  this  same  model  to  a  fleet-selection  problem  and  make  it  more  tractable  via 
variable-elimination  techniques. 

An  optimal  requirements  airlift  mobility  study  is  discussed  in  Yang  et  al.  (1996).  The 
authors  formulate  the  mobility  system  as  a  pickup  and  delivery  vehicle  routing  and  scheduling 
problem  with  time -window  constraints.  After  developing  a  set-partitioning  based  formulation, 
the  authors  apply  column-generation  and  column-elimination  heuristics  to  decrease  the 
computational  complexity  of  their  mathematical  model. 
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Baker  et  al.  (2002)  provide  an  extensive  discussion  on  the  history  of  mathematical 
models  within  airlift  mobility.  The  NRMO  model  presented  in  Baker  et  al.  (2002)  captures  time- 
dynamic  movement  of  large-scale  military  deployment  operations  and  considers  aerial  refueling, 
crew  availability,  intra-continental  aircraft  shuttles,  and  unit  resolution  for  cargo.  Finally,  deNijs 
(2004)  develops  an  integer  programming  formulation  of  the  airlift  problem.  The  model’s 
objective  is  to  minimize  the  number  of  ship  sailings  and  aircraft  sorties  (the  number  of  ships  and 
aircraft  allocated  to  the  deployment  operation). 

1.3.2  Previous  Heuristic-Based  Optimization  Research 

In  addition  to  mathematical  programming-based  methodologies,  a  number  of  authors 
have  investigated  heuristic  solution  methodologies  for  air  mobility  operations.  McKinzie  and 
Barnes  (2004)  review  Strategic  Mobility  Models  (SMMs)  from  the  viewpoint  of  strategic 
logistics  and  force  protection.  The  authors  provide  an  extensive  review  of  force  deployment 
practices  within  and  outside  CONUS,  and  describe  cargo  and  passenger  and  passenger  flow 
during  deployment. 

Crino  et  al.  (2004)  investigated  the  theater-distribution  vehicle  routing  and  scheduling 
problem  using  group-theoretic  tabu  search.  In  their  study,  the  authors  describe  both  the 
conceptual  context,  based  on  a  flexible  group-theoretic  tabu  search  framework,  and  the  software 
implementation  of  a  robust,  efficient,  and  effective,  prescriptive  generalized-theater-distribution 
methodology.  Their  methodology  evaluates  and  prescribes  the  routing  and  scheduling  of 
multimodal  theater  transportation  assets  with  the  objective  of  minimizing  unmet  customer 
demand,  late  deliveries,  and  vehicle  costs  (fixed  and  variable).  Weber  and  Bojduj  (2006)  also 
utilize  a  tabu  search  algorithm  for  simultaneously  solving  strategic  and  operational  levels  of 
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planning.  Their  algorithm  uses  partial  order  planning  to  separate  the  optimization  process  from 
the  constraint  verification  process. 

Godfrey  et  al.  (2004)  present  negotiation  protocols  for  the  distributed  optimization 
problem  of  commercial  air  carriers  supporting  military  airlifts.  Their  approach  to  the  airlift 
planning  and  scheduling  problem  uses  software  agents  to  collaboratively  plan  the  airlift.  The 
current  approach  is  auctioning  the  missions  subject  to  a  reserved  price  and  allowing  carriers  to 
swap  missions  when  mutually  beneficial.  Their  study  cuts  the  controllable  operating  costs  and 
schedule-disruption  costs  by  more  than  half  compared  with  the  centralized  planning  approach 
currently  used.  Furthermore,  their  approach  protects  the  military  by  capping  mission  profit 
potential  using  a  Vickrey  auction  mechanism  (deVries  and  Vohra,  2003),  and  protects  the 
carriers  from  being  forced  to  share  information  or  cooperate  with  its  economic  competitors. 
Smith  et  al.  (2004)  describe  an  incremental,  constraint-based  scheduling  model  that  provides 
support  for  day-to-day  allocation  and  management  for  barrel  allocation  problems.  The  presented 
system  in  that  study  utilizes  search  procedures  to  provide  a  range  of  automated  and  semi- 
automated  scheduling  capabilities. 

Clark  et  al.  (2004)  present  a  large-scale  optimization-based  planning  method  that  uses 
column  generation  to  schedule  the  movement  of  ammunition  and  transportation  resources 
through  a  time-space  network  representation  of  the  distribution  system.  In  their  method,  the 
optimization-based  planner  is  initialized  using  a  feasible  solution  generated  by  a  heuristic 
planning  method.  Both  the  optimization-based  planner  and  the  heuristic  planner  generate  plans 
with  improved  vehicle  utilization  and  delivery  tardiness  values  as  compared  to  plans  generated 
using  current  planning  techniques.  In  addition,  the  heuristic  planner  is  implemented  within  a 
closed-loop  planning  and  control  framework  and  is  used  to  generate  plans  on  a  rolling  horizon 
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basis.  Johnstone  et  al.  (2004)  developed  a  mixed  integer  program  model  of  munitions  flow  with 
the  primary  objective  of  minimizing  the  overall  response  time  to  meet  demands  of  the  various  air 
bases.  They  also  demonstrate  an  analysis  methodology  using  their  model  by  implementing  it  in 
an  algebraic  modeling  environment,  employing  a  standard  general-purpose  solution  routine,  and 
verifying  against  a  realistic  planning  scenario. 

Although  most  of  the  abovementioned  research  focuses  on  creating  mathematically  sound 
optimization  models,  the  research  does  not  take  dependency  and  precedence  constraints  into 
account  simultaneously.  Besides,  some  network  flow  and  simulation  models  are  unable  to 
provide  timely  results,  while  some  mathematical  models  lack  capturing  the  overall  system 
behavior  due  to  dynamic  and  non-hierarchical  structure-of-problem  domain.  In  this  study,  we 
present  a  mathematical  model  for  item-to-aircraft  assignment  and  scheduling  of  air  missions  for 
military  disaster  and  humanitarian  relief  operations.  For  making  assignment  and  scheduling 
decisions,  dependency  and  precedence  constraints  are  considered.  Aircraft  are  the  only 
transportation  mode  used  in  this  study,  and  every  aircraft  is  assumed  to  have  passenger  and  cargo 
weight  and  volume  capacity.  Performance  of  the  model  is  evaluated  via  test  cases  using  real  data. 

1 .4  Mathematical  Programming  Model 

The  following  mathematical  model  takes  item-dependent  constraints  into  account.  As  of 
our  knowledge,  this  is  the  first  use  of  these  constraints  in  the  military  context.  The  constraints 
included  in  this  model  are  dependability  and  precedence  of  the  items  shipped.  The  main  idea  is 
that  if  different  types  of  loads  are  not  considered  in  deployment  operation,  inefficiencies  will 
occur.  Examples  can  be  given  as  military  equipment  would  not  be  useful  without  the  soldiers, 


transporting  medical  equipment  with  construction  equipment  might  be  hannful  to  some  types  of 
medicine. 

There  are  certain  characteristics  of  the  model.  Only  one  depot  and  one  contingency 
location  are  considered.  A  load  consists  of  passengers  and  cargo  items.  Depending  on  the 
situation,  different  modes  of  transportation  are  considered  with  different  capacities  related  to 
passengers  and  cargo  items.  This  model  takes  the  cumulative  multi-period  demand,  aircraft 
availability,  aircraft  capacity,  weights  and  volumes  of  the  cargo  items  as  parameters  and  assigns 
the  loads  to  available  aircrafts  while  meeting  scheduling  requirements.  Aerial  refueling,  crew 
availability,  maximum-on- the-ground  constraints  are  excluded  from  the  model. 


1.4.1  Sets 

I  set  of  first  type  of  containers 

J  set  of  second  type  of  containers 
R  set  of  all  containers  where  R  =  I  u  J 
P  set  of  passengers 

C  set  of  cargo  items 

N  set  of  all  load  where  N  =  PuC 
T  set  of  periods 

1.4.2  Parameters 

Wj  weight  capacity  of  aircraft  j  (pounds)  for  V/  e  J 
Wi  weight  of  an  cargo  item  i  (pounds)  for  VieC 
Vj  volume  capacity  of  aircraft  j  (cubic  feet)  for  V/  e  J 
Vi  volume  of  an  cargo  item  i  (cubic  feet)  for  VieC 

Qj  passenger  capacity  of  aircraft  j  (quantity)  for  V/  e  J 

D!  demand  of  item  i  (quantity)  at  the  contingency  by  period  t 

aj  round  trip  travel  time  (periods)  for  V/  e  J 

B  nonnalizing  value  for  percent  demand  satisfied 

rf  cumulative  availability  of  item  /(quantity)  at  the  supply  base  by  period  t 

b  nxt 7 binary  matrix  that  shows  the  compatibility  relationship  between  items 
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1.4.4  Model 
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The  model’s  objective  function  (1)  is  to  minimize  the  nonnalized  total  weighted  earliness 
and  tardiness.  Constraint  set  (2)  ensures  that  cargo  weight  capacity  of  an  aircraft  is  not  violated. 
Constraint  set  (3)  ensures  that  passenger  capacity  of  an  aircraft  is  not  violated.  Constraint  set  (4) 
ensures  that  the  cargo  volume  capacity  of  an  aircraft  is  not  violated.  Constraint  set  (5)  makes 
sure  that  the  availability  of  aircraft  at  the  depot.  Constraint  set  (6)  links  the  availability  and 
assignment  decisions.  Constraint  set  (7)  links  the  availability  and  assignment  decisions. 
Constraint  set  (8)  assigns  early  and  late  items.  Constraint  set  (9)  verifies  that  the  items  are 
released  before  loaded  on  aircraft.  Constraint  set  (10)  satisfies  the  compatibility  between  items 
(mutual  exclusiveness).  Constraint  set  (11)  sets  the  volume  utilization  of  an  aircraft.  Constraint 
set  (12)  sets  the  weight  utilization  of  an  aircraft.  Finally,  constraint  set  (13)  sets  the  time 
utilization  over  all  periods. 
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1 .5  Experimental  Scenario  Development 

Our  mathematical  model  is  tested  with  realistic  test  scenarios  where  different  demands 
are  required  for  different  scenarios.  We  took  into  consideration  three  different  scenarios  that  our 
model  could  be  used.  The  purpose  of  using  different  scenarios  such  as  a  landslide,  flood,  and 
tsunami  disaster  is  to  prove  that  our  model  could  be  flexible  to  handle  different  types  of  demand 
with  different  time  constraints  using  various  modes  of  transportation.  It  is  vital  to  us  that  it  could 
all  be  changed  to  what  the  user  considers  to  be  the  realistic  for  different  situations. 

Generally,  the  types  of  cargo  items  are  living,  medical,  and  military  equipments,  food, 
and  clothing.  Types  of  passengers  are  soldiers,  doctors,  and  workers.  Medical  equipment  is 
assumed  to  be  a  container  full  of  medical  supplies  for  one  doctor,  construction  equipment  is 
assumed  to  be  a  midsize  Bobcat  excavator.  Food  is  Meal,  Ready  to  Eat  (MRE)  and  military 
equipment  is  considered  to  be  a  foot  locker  of  needed  weapons  for  one  soldier. 

The  release  date  matrix  shows  the  demand  quantity  in  certain  mission  points  and  weights 
associated  with  different  types  of  loads.  The  amount  released  was  detennined  using  the  product 
of  the  maximum  total  demand  scaled  by  a  scalar  of  1 .05  and  gamma(2,2)  distribution.  The  scalar 
was  used  to  elongate  the  time,  in  periods,  that  the  maximum  demand  for  each  item  was  met 
halfway.  Further,  the  importance  of  items  over  time  depended  on  the  scenario.  Our  information 
was  received  through  interviews  with  retired  military  experts. 

The  earliness  and  tardiness  values  are  nonnalized  by  multiplying  the  value  by  the  ratio  of 
the  total  demand  of  the  greatest  numerically  demanded  item  to  the  total  demand  of  the  item  in 
question.  This  places  the  item  in  question’s  demand  in  the  units  of  the  greatest  numerically 
demanded  item,  allowing  them  to  be  proportionately  added  in  the  objective  function.  The  values 
are  further  weighted  by  the  time-dependent  importance. 
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1.5.1  Landslide  Scenario 


First,  a  landslide  scenario  is  tested.  Specific  numbers  of  aircraft  are  assumed  to  be 
available  at  the  beginning  of  deployment.  The  aircraft  used  are  shown  below  in  Table  1.  The 
time  horizon  is  23  periods,  where  a  period  is  assumed  to  be  12  hours.  The  demand  is  ordered 
according  to  the  time  periods.  Along  with  doctors,  medical  and  living  equipment,  food,  and 
construction  workers,  we  need  heavier  machinery  and  utility  workers  as  well.  The  compatibility 
relationships  were  detennined  to  not  allow  the  heavy  machinery  to  be  loaded  on  the  same  aircraft 
as  the  food/water  and  medical  equipment  due  to  the  possible  damage  and/or  contamination  that 
the  machinery  can  inflict.  According  to  retired  military  experts  one  of  the  main  tasks  in  a 
landslide  is  to  uncover  the  bodies.  Therefore  we  send  construction  equipment  and  workers  first, 
then  the  doctors  and  medical  and  living  equipment  for  the  survivors,  as  landslide  response 
consists  of  uncovering  bodies  then  dealing  with  infrastructure  and  transportation. 


Table  1.  Aircraft  Type  Characteristic  Data 


Characteristic  name 

Aircraft  Type 
C-17  C-5 

cargo  capacity  (lb) 

170,000 

400,000 

passenger  capacity 

102 

73 

speed  (miles/hr) 
travel  time  to  contingency 

470 

330 

(periods) 

2 

3 

volume  capacity(ft3) 

24,000 

42,000 

Our  mixed-integer  program  is  coded  in  AMPL  and  analyzed  using  CPLEX  vlO.l.  The 
test  case  scenarios  are  run  on  a  quad  core,  quad  processor  PC-based  server  with  128  GB  of 
RAM.  All  runs  produced  feasible  results.  Aircraft  are  used  multiple  times  with  round  trips  during 
the  time  horizon.  The  longest  solve  time  was  35.9  minutes.  The  average  solve  time  was  7.6 
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minutes.  The  airplane  usage  is  shown  in  Table  2.  Further,  Tables  3  and  4  show  example  load 
information  for  aircraft  Al. 


Table  2.  Landslide  Aircraft  Usage 


Table  3.  Landslide  Load  Distribution  Airplane  Al 
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Table  4.  Landslide  Weight  Distribution  Airplane  A1 


Percent  of  Weight  Loaded 


■  Med  Equip 

■  Doctors 

■  Heavy  Mach 

■  Util  Worker 

■  Living  Equip 

■  Food 
Living  Equip 


1.5.2  Flood  Scenario 

For  this  scenario  we  assume  that  helicopters  would  also  be  used  due  to  the  area  that  needs 
the  relief  will  most  likely  be  unusable  for  a  cargo  transporter  plane  to  land  safely.  The  data  for 
the  helicopters  used  are  explained  in  Table  5.  The  compatibility  relationships  were  issued  to  be 
that  the  food/water  and  living  equipment  could  not  be  loaded  on  the  same  aircraft.  Living 
equipment  is  assumed  to  be  generators  that  carry  gas  in  which  it  might  be  considered  harmful  to 
the  food/water  if  some  were  damaged  during  transport. 


Table  5.  Aircraft  Type  Characteristic  Data 


Aircraft  Type 


Characteristic  name 


HH  -60  G  CH  -  47 

Pave  Hawk  Chinook 


cargo  capacity  (lb) 

6000 

28000 

passenger  capacity 

10 

40 

speed  (miles/hr) 

184 

137 

travel  time  to  contingency  (periods) 

2 

3 

volume  capacity(ft3) 

416 

1400 

15 


In  terms  of  our  experimental  results,  helicopters  are  used  multiple  times  with  round  trips 
during  the  time  horizon.  The  longest  solve  time  was  11.1  minutes.  The  average  solve  time  was 
3.1  minutes.  The  helicopter  usage  is  shown  in  Table  6.  Further,  Tables  7  and  8  show  the  load 
information  for  aircraft  Al. 


Table  6.  Flood  Helicopter  Usage 
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Table  7.  Flood  Load  Distribution  Helicopter  A1 
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Table  8.  Flood  Weight  Distribution  Helicopter  A 1 
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1.5.3  Tsunami  Scenario 

We  used  the  same  plane  assumptions  as  the  landslide  scenario  shown  in  Table  1  for  our 
tsunami  scenario  investigations.  The  compatibility  relationships  between  the  items  are  that 
medical  equipment  cannot  be  loaded  into  the  same  aircraft  as  construction  or  military  equipment 
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as  well  as  military  equipment  cannot  be  loaded  on  the  same  aircraft  as  food/water  because  the 
food/water  and  medical  equipment  could  be  damaged  or  contaminated  during  flight.  We  added 
soldiers  and  military  equipment  to  the  demand  to  guard  and  control  the  area  of  the  disaster 
location.  Living  equipment,  in  this  case,  is  considered  to  be  tents  and  their  contents. 

In  this  final  scenario,  aircraft  are  used  multiple  times  with  round  trips  during  the  time 
horizon.  The  longest  solve  time  was  46.5  minutes.  The  average  solve  time  was  6.1  minutes.  The 
airplane  usage  is  shown  in  Table  9.  Finally,  Tables  10  and  11  show  the  load  infonnation  for 
aircraft  Al. 


Table  9.  Tsunami  Aircraft  Usage 
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Table  10.  Landslide  Load  Distribution  Airplane  A 1 


Percent  of  Units  Loaded 


■  Doctors 

■  Workers 

■  Soldiers 

■  Med  Equip 

■  Heavy  Mach 

■  Food 
Clothing 

■  Military  Equip 
Living  Equip 


Table  11.  Landslide  Weight  Distribution  Airplane  A 1 
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1 .6  Conclusions  and  Future  Research 


The  problem  of  item-to-aircraft  assignment  and  scheduling  of  air  missions  for  military 
disaster  and  humanitarian  relief  operations  is  analyzed  in  this  paper.  Item-specific  dependency 
and  precedence  constraints  are  embedded  in  a  Mixed  Integer  Programming  (MIP)  formulation 
with  time-varying  demands  as  critical  periods.  The  MIP  formulation  is  tested  via  realistic 
scenarios  and  is  shown  to  be  functioning  correctly.  As  of  authors’  knowledge,  this  is  the  first  use 
of  item-specific  dependency  and  precedence  constraints  simultaneously  in  a  military  context. 

The  computational  results  point  toward  more  thorough  analysis  of  tractability  of  MIP 
formulation.  Although  results  are  obtained  for  six  and  eight  aircraft,  ten  aircraft  case  couldn’t  be 
solved  by  MIP  formulation  in  a  reasonable  amount  of  time.  The  formulation  should  be  improved 
to  get  better  cuts  during  MIP  branching.  Our  optimization  model  is  capable  of  accommodating 
different  types  of  transportation  assets  with  varying  capacities  (e.g.,  airplane  vs.  helicopter). 
Experimental  results  show  different  vehicle  loading  profiles  corresponding  to  item-type  due 
dates,  compatibilities,  and  precedence  relationships. 

Look-ahead  scheduling,  improvement  in  plane  packing,  and  combing  unit  types  might  be 
studied  in  future  work  to  get  more  realistic  fonnulations,  along  with  Aerial  refueling,  crew 
availability,  and  maximum-on- the-ground  constraints.  The  MIP  fonnulation  developed  in  this 
paper  can  be  used  in  the  previous  literature  (Baker  et  al.,  2002;  Rosenthal  et  ah,  1997).  This  will 
be  a  significant  improvement  for  the  current  models. 
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2.1  Executive  Summary 

Team-based  work  structures  have  increasingly  adopted  to  address  a  variety  of  complex 
and  difficult  logistics  tasks  due  to  substantial  benefits  associated  with  teamwork.  The  reality  is, 
however,  that  there  is  little  guarantee  of  success  as  many  teams  fail  for  any  number  of  reasons. 
This  is  the  case  especially  when  a  distributed  logistics  team  must  make  decisions  with  severe 
consequences,  while  coordinating  and  collaborating  with  other  parties  involved.  Thus,  it  is 
critical  to  have  a  full  understanding  of  the  factors  affecting  team  effectiveness  that  perform 
logistics  operations  which  require  additional  attention,  intensive  management,  and  continuous 
monitoring.  However,  there  is  a  lack  of  empirical  evidence  on  factors  that  may  affect  team 
performance  and  cognition,  including  the  complex  cognitive  factors  driving  the  behavioral 
phenomena  associated  with  the  distributed  logistics  teams.  The  main  focus  of  the  project  was  to 
provide  experimental  evidence  on  team  coordination  and  collaboration  within  a  distributed 
logistics  network.  Relatively  few  empirical  research  studies  have  been  conducted  on  how 
geographically  dispersed  logistics  teams  coordinate  their  logistics  activities  and  collaborate  with 
other  involved  parties  to  perform  logistics  operations  in  a  joint  environment  (e.g.,  aerial  port 
simulation). 
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2.2  Introduction 


Much  of  the  work  in  organizations  today  is  completed  through  team  based  work 
structures,  aimed  at  facilitating  team  member  interactions  in  an  effort  to  promote  successful  task 
completion  (Cannon-Bowers  et  ah,  1993),  which  has  proved  to  increase  productivity  and 
expertise  as  well  as  minimize  workloads  for  individuals  (Marks  et  ah,  2001;  O’Connora  et  ah, 
2008).  The  reality  is,  however,  that  many  teams  tend  to  fail  for  any  number  of  reasons 
(Hackman,  1998). 

Existing  studies  on  team  effectiveness  fall  far  short  of  providing  practical  guidance  for 
managing  distributed  logistics  teams  for  at  least  three  reasons.  First,  many  aspects  of  team 
coordination  and  collaboration  in  distributed  logistics  contexts  still  remain  unclear,  because  team 
effectiveness  has  been  detennined  mainly  by  team  performance  .  However,  recent  evidence 
supports  the  assertion  that  common  cognitions  among  team  members  and  their  effective 
communication  process  are  closely  associated  with  team  effectiveness.  Thus,  team  cognition  and 
team  communication  should  also  be  considered  to  fully  uncover  the  various  aspects  of 
distributed  logistic  teams  (Levi  and  Slem,  1995;  Littlejohn,  2002;  May  and  Carter,  2001). 

Second,  relatively  few  studies  have  been  carried  out  to  systematically  investigate  the  influence  of 
the  factors  that  may  facilitate  or  hinder  team  coordination  and  collaboration  in  logistics  contexts 
(Nam  and  Thomas,  2006;  Nam  et  ah,  2009).  Finally,  only  a  few  simulation  platforms  are  still 
available  for  researchers  to  use  as  a  test-bed  in  order  to  perform  collaborative  logistics  research 
(Nam  et  ah,  2007).  The  research  platform  is  required,  which  allows  researchers  to  manipulate 
and  investigate  a  variety  of  logistics  scenarios  (e.g.,  inclusion  of  unexpected  events  such  as 
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diverted  mission  and  transportation  system  breaking  down  to  diagnose  team-based  planning, 
coordination,  and  collaboration). 

The  goal  of  the  present  study  was  to  provide  experimental  evidence  on  team  coordination  and 
collaboration  within  a  distributed  logistics  network  by  conducting  an  empirical  experiment  on 
factors  that  may  facilitate  or  hinder  team  effectiveness  in  logistics  contexts.  Specifically,  the 
objectives  are: 

1.  To  evaluate  the  overall  effectiveness  of  a  computer-based  platform  for  collaborative 
logistics  research,  Computer-Based  Aerial  Port  Simulation  (CAPS)  system; 

2.  To  manipulate  and  empirically  investigate  two  contributing  factors  such  as  task  workload 
and  communication  availability;  and 

3.  To  uncover  comprehensive  aspects  of  team  effectiveness  in  distributed  logistics  contexts 
by  collectively  measuring  team  perfonnance  (e.g.,  departure  time)  and  team  cognition 
(e.g.,  team  mental  model  and  team  interaction  process). 

2.3  Participants 

Fifty  participants  from  the  University  of  Arkansas  student  population  were  recruited  for 
the  experiment.  Participants  were  given  extra  credit  for  compensation  for  their  participation. 
There  were  38  male  and  12  female  participants  with  a  mean  age  of  23.4  years  ( SD  =  3.02  years). 
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2.4  Experimental  Task  and  Apparatus 

A  computer-based  laboratory  task  that  requires  collaboration  within  a  distributed  logistics 
network,  Computer-Based  Aerial  Port  Simulation  (CAPS),  was  developed  in  collaboration  with 
Dr.  Joseph  Lyons  of  Air  Force  Research  Laboratory/Logistics  Readiness  Branch  (AFRL/HEAL). 
Participants  were  asked  to  perform  an  aerial  port  task  as  a  team  in  CAPS,  in  which  they  needed 
to  provide  air  transportation  to  all  users  of  the  Defense  Transportation  System  (DTS)  in  the 
timely  manner. 


2.4.1  Computer-Based  Aerial  Port  Simulation  (CAPS): 

CAPS  enables  a  team  of  five  interdependent  participants  to  simulate  logistics  operations 
associated  with  an  aerial  port  squadron  (e.g.,  movement  of  air  cargo  and  passengers  through  the 
Defense  Transportation  System).  Aerial  ports  consist  of  five  primary  functional  sections:  the  air 
tenninal  operations  flight  (ATOF),  passenger  services  (PS),  fleet  services  (FS),  cargo  services 
(CS),  and  ramp  services  (RS)  (see  Appendix  1  for  critical  tasks  to  be  performed  at  each 
functional  section  in  CAPS). 

Table  12.  An  Example  of  Task  Scenario 

ATOF  receives  information  from  command  post  about  an  inbound  C-17  originating  from  Dover  AFB  in 
transit  to  Travis  AFB.  The  aircraft  will  be  arrived  in  approximately  25  minutes.  The  C-17  will  need  to 
download  5  terminating  pallets  and  upload  7  pallets  in  route  to  Travis  AFB.  Twenty-five  passengers  will 
need  to  be  downloaded  from  the  aircraft  and  12  will  need  to  be  boarded  in  route  to  Travis  AFB.  Each 
passenger  requires  an  in-flight  meal.  Once  the  C-17  has  landed,  the  aerial  port  will  have  3  hours  to  complete 
port  operations  and  clear  the  aircraft  from  the  port.  Once  cleared  from  the  port,  the  aircraft  can  take  off. 
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One  team  member  represents  one  section  within  the  aerial  port  during  the  CAPS  scenario. 
Team  members  communicate  using  a  chat  system,  and  all  chat  data  were  stored  in  an  automated 
database  for  subsequent  coding  and  analysis.  Unexpected  events,  such  as  diverted  missions  and 
aircraft  malfunctions,  can  be  injected  into  the  scenario  to  diagnose  team-based  problem-solving, 
coordination,  and  collaboration.  Figure  1  shows  a  sample  task  scenario  used  in  the  experiment. 

(1)  Air  Terminal  Operations  Flight  (ATOF):  As  shown  in  Figure  2,  the  ATOF  provides 
the  aerial  port  sections  with  information  to  manage  available  resources  to  receive,  document,  and 
move  passengers  and  cargo.  It  is  the  focal  point  through  which  all  infonnation  relating  to  airlift 
flow  is  received,  processed,  and  dispatched  to  functional  areas.  All  functional  sections  will  need 
to  be  notified  when  the  aircraft  has  landed.  ATOF  should  remind  the  sections  about  what  the 
aircraft  needs  (e.g.,  cargo  &  passenger  requirements).  All  sections  will  need  to  be  monitored  to 
insure  that  the  aircraft  is  not  delayed.  ATOF  will  trouble  shoot  any  problems  during  the  port 
operations  (e.g.,  vehicle  maintenance  will  need  to  be  notified  if  a  vehicle  breaks  down,  or  all 
sections  will  need  to  be  notified  if  an  aircraft’s  status  changes).  As  soon  as  all  functional  sections 
have  completed  their  duties,  ATOF  can  port  clear  the  aircraft  and  it  can  depart. 
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Figure  1.  Communication  Diagram  of  the  Air  Tenninal  Operations  Flight  (ATOF) 

(2)  Ramp  Services  (RS):  Ramp  services  load  out-bound  cargo  and  download  in-bound 
palletized  cargo  for  the  aerial  port.  Ramp  will  need  to  download  the  necessary  pallets  and  return 
them  to  the  cargo  section  for  processing.  They  will  also  need  to  load  the  necessary  pallets  onto 
the  C-17  that  are  required  to  be  transported  to  Travis  AFB. 

(3)  Passenger  Services  (PS):  Passenger  services  process,  embark,  and  debark  all  passengers 
in  the  aerial  port  (see  Figure  2).  This  section  will  need  to  download  the  passengers  coming  in 
from  Dover  and  return  them  to  the  passenger  terminal  for  processing.  They  will  also  need  to 
upload  any  passengers  going  to  Travis.  They  must  coordinate  with  Fleet  services  to  get  the 
appropriate  number  of  passenger  meals  ordered  and  delivered  to  the  aircraft. 
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Figure  2.  An  Example  of  Passenger  Services  Interface 


(4)  Fleet  services  (FS):  This  section  supplies  transport  aircraft  with  passenger  and  crew 
comfort  items  (including  meals)  while  ensuring  that  transport  aircraft  interiors  are  clean  and 
presentable. 

(5)  Cargo  services  (CS):  This  section  must  prioritize  cargo  by  destination:  where  the  cargo 
is  going,  and  priority:  some  cargo  must  be  transported  before  others.  Once  information  is 
received  about  what  an  aircraft  needs,  this  section  must  sequence  loads  so  that  ramp  can  pick 
them  up.  This  section  must  also  process  all  incoming  cargo. 


2.5  Experimental  Design 

There  were  two  independent  variables  manipulated  in  the  study:  (1)  workload  (low  vs. 
high)  and  (2)  communication  availability  (partly  and  indirectly  vs.  all  and  directly).  The 
workload  factor  was  a  within-subject  variable  and  the  communication  availability  was  a 
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between-subject  variable.  As  shown  in  Table  1,  a  2  (workload)  x  2  (communication  availability) 
mixed-factor  design  was  used  in  the  study. 


Table  13.  Mixed  Factor  Experimental  Design  Used  in  the  Study 


Workload  (within-subject  variable) 

Low(Session  1) 

High(  Session  1) 

Communication 

All  and 

Team  1 

Team  1 

Availability 

directly 

Team  5 

Team  5 

(between-subject 

Partly 

Team  6 

Team  6 

variable) 

& 

indirectly 

Team  10 

Team  10 

Workload:  Workload  variable  was  manipulated  by  altering  the  number  of  air  craft  per  session 
and  the  time  demands  under  which  team  members  must  operate  (e.g.,  arrival  and  departure 
times). 

Communication  Availability:  Communication  availability  was  manipulated  either  all 
communication  channels  open  (i.e.,  all  and  direct  communication  among  all  the  team  members) 
or  only  individual  chat  capability  available  (i.e.,  communication  among  team  members  is  limited 
according  to  the  predefined  communication  protocol). 
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2.6  Procedures 


Participants  were  randomly  assigned  to  1  of  10  work  groups  with  five  members,  in  which 
they  remained  in  the  same  group  for  the  duration  of  the  study.  The  participants  were  asked  to 
come  to  the  Human-Computer  Interaction  (HCI)  lab  in  the  Department  of  Industrial  Engineering 
at  the  University  of  Arkansas  for  the  experiment  lasting  about  two  hours  to  complete.  Team 
members  were  asked  to  perfonn  an  aerial  port  task  in  CAPS  (Computer-based  Aerial  Port 
Simulation),  in  which  they  needed  to  provide  air  transportation  to  all  users  of  the  Defense 
Transportation  System  (DTS)  in  the  timely  manner.  Two  sessions  were  conducted  in  the 
experiment  and  the  entire  session  was  video-  and  audiotaped  for  later  examination.  After  the 
each  session,  the  participants  were  also  asked  to  complete  a  questionnaire  designed  to  investigate 
team  cognition  (e.g.,  team  mental  model). 

Before  the  experiment,  the  participants  were  asked  to  complete  a  background  survey 
questionnaire  such  as  personality  and  education  level.  The  participants  also  performed  the 
training  exercise  to  get  themselves  familiarize  with  the  functional  section  they  were  assigned. 
Figure  3  demonstrates  an  example  of  the  training  material  designed  to  train  the  Passenger 
Services  section. 
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Passenger  Services  (PS)  Primary  Tasks: 


1-  Unload  (Disembark) passengers  _ 

•  Right-click  on  tlie  bus  w  and  select  Move  to  Ramp  #'  U/  from  the  menu. 

•  Once  the  bus  is  at  the  aircraft,  selec:  Download  Pasengen"  J&troin  the  menu. 

•  Right-click  again  on  the  bus  and  select  Return  to  PS  Bldg1.;*;  horn  the  menu 

•  Pugh: -click  ou  the  bus  and  select  "Disembark  Passengers  vg>  from  the  menu. 

•  Notify  ATOF  and  Ramp  Services  via  IM  C  hat  immedtatny  after  you  have  unloaded  the  passengers. 


2.  Iu-Process  Passengers 

Before  passengers  can  fly  on  military  flights,  they  must  be  entered  into  ±e  system. 

Use  passengers  social  security'  numbers  (SSN)  A)  from  their  Travel  Request  (TR  ^j  to  retrieve  information  about  the  passenger. 
Assign  passengers  to  a  gate  based  on  then  desireaaestxatiou  (eg.  Germany). 

It  is  important  that  you  assign  all  passengers  heading  to  the  same  destination  to  the  same  gate. 

•  Right  dick  on  the  ?S  Terminal  M  and  select  In-Process  Passengers  '  from  the  menu. 

•  A  Travel  Request  ©will  ann|arLocate  the  SSN'©on  the  TR  and  type  it  into  the  SSN  text  box®  in  the  in-processing  section 

•  Press  the  "Search"  button  Q)  The  Name,  Ranh  Grade  and  Service  of  the  passenger  will  appear  x  the  form. 

•  Based  on  the  navel  request,  select  the  correct  number  of  Seats  and  the  correct  Travel  Status  in  the  form. 

•  Make  selections  for  the  Destination  Country  and  the  Passenger  Gate  Number  in  the  form. 

•  Click  on  the  Submit'  button®  or  select  another  Passenger  via  the  "Next  »>”  button  ®  to  continue  working 

3.  Load  Passengers 

•  Select  “Load  Passengers'  from  the  bus  menu  and  enter  the  correct  Gate  Number  (he.,  where  the  passengers  are  waiting). 

•  Select  "Move  to  Ramp  »"  from  the  bus  menu  to  move  the  bus  to  the  appropriate  aircraft. 

•  Select  "Upload  Passengers"  from  the  bus  menu 

•  Notify  both  ATOF  and  Fleet  Services  via  IM  C  hat  immediately  after  yon  have  loaded  the  passengers. 


Figure  3.  An  Example  of  Training  Materials  (for  passenger  services  section) 
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2.7  Results  and  Discussion 


To  empirically  investigate  the  effects  of  task  workload  and  communication  availability  on 
team  effectiveness  in  a  distributed  logistics  context,  this  study  used  several  dependent  measures, 
which  can  be  categorized  into  two  characteristics  of  variables:  team  performance  and  team 
cognition.  Two-way  analyses  of  variances  (ANOVAs)  will  be  perfonned  to  detennine  the  effects 
of  workload  and  communication  availability  on  task  performance  and  team  cognition.  Since  the 
experiment  is  still  being  performed  and  the  data  collection  has  not  been  completed  yet,  the  PI 
will  briefly  describe  the  plan  of  the  data  analysis  and  the  expected  results  in  this  section. 

2. 7. 1  Team  Performance 

(1)  Success/Failure  of  Task:  Team’s  perfonnance  will  be  assessed  on  the  basis  of  whether 
the  aircraft  left  on  time. 

(2)  Performance  Score:  Three  types  of  scores  will  be  utilized,  which  are  provided  by  CAPS 
system. 

General  Score:  General  score  is  the  score  a  station  can  get  by  performing  actions  that  are 
not  directly  related  to  a  specific  aircraft.  For  example,  Passenger  Services  in-processing 
passengers  is  a  action  that  does  not  directly  affect  an  aircraft  so  that  action  would  be 
recorded  under  the  ’General  Score'  section. 

-  AC#  Score:  AC#  score  where  #  is  the  aircraft  number  is  modified  by  any  action  that 
directly  affects  an  aircraft.  For  example,  Passenger  Services  loading  passengers  onto  an 
aircraft  will  be  recorded  under  the  ’AC#  Score’  section  where  x  is  the  aircraft  number. 
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-  All  Score:  All  score  value  is  sums  the  performance  scores  across  aircraft,  general  score, 
and  station/team  member 

2.7.2  Team  Cognition 

(1)  Stressor  Appraisal  Scale  (SAS)  (Schneider,  2008):  Ten  items  assessed  stressor 
appraisals.  Primary  appraisal  items  included:  (1)  how  threatening  do  you  expect  the 
upcoming  task  to  be;  (2)  how  demanding  do  you  think  the  upcoming  task  will  be;  (3) 
how  stressful  do  you  expect  the  upcoming  task  to  be;  (4)  to  what  extent  do  you  think  you 
will  need  to  exert  yourself  to  deal  with  this  task;  (5)  how  much  effort  (mental  or 
physical)  do  you  think  the  situation  will  require  you  to  expend;  (6)  how  important  is  it 
for  you  to  do  well  on  this  task;  and  (7)  how  uncertain  are  you  about  what  will  happen 
during  this  task.  Secondary  appraisal  items  included:  (1)  how  well  do  you  think  you  can 
manage  the  demands  imposed  on  you  by  this  task;  (2)  how  able  are  you  to  cope  with  this 
task;  and  (3)  how  well  do  you  think  you  will  perform  this  task.  Items  were  rated  on 
seven-point  Likert  scales. 

(2)  Interpersonal  Trust  (Naquin  &  Paulson,  2003):  To  measure  trust  level  among  team 
members,  Naquin  &  Paulson’s  (2003)  interpersonal  trust  questionnaire  will  be  used  (see 
Appendix  2). 

(3)  Shared  Team  Mental  Model:  Kendall’s  coefficient  of  concordance  will  be  used  to 
investigate  the  mental  model  shared  by  team  members  (Lewis  and  Johnson,  1971). 
Kendall  (1955)  presented  a  Coefficient  of  Concordance,  W,  to  evaluate  the  extent  of 
agreement  among  a  set  of  judges  each  of  whom  ranks  in  entirety  a  set  of  objects;  this 
statistic  is  well  known  and  has  been  widely  used  in  the  sociometric  literature. 
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Let  R  be  an  n  x  m  matrix  in  which  rM  is  the  rank  of  the  j-th  of  m  objects  as  judged  by  the  i-th 
of  n  judges.  Then 

W  =  12S/m2n  (n2  —  1) 

where  S  is  the  sum  of  the  squared  deviations  of  the  column  sums  from  the  mean  column  sum, 

S  =  5(r  j  —  r../m)2 

In  the  case  where  the  objects  to  be  ranked  are  the  judges  themselves,  n=m  and 

W=  12S/n3  (n2-  1) 

As  applied  to  ranking  matrices,  Youden  arrays  are  characterized  by  the  fact  that  each  pair  of 
objects  to  be  ranked  appears  the  same  number  of  times.  In  this  case 

W  =  12S/A2n  (n2  —  1) 

where  X  is  the  number  of  times  that  a  given  comparison  occurs.  When  each  member  of  a 
group  tanks  all  of  the  members  except  himself,  each  pair  of  members  is  ranked  n-2  times.  This  is 
obvious  from  the  fact  that  the  (i,  j)  pair  is  omitted  by  person  i  and  person  j  but  by  no  one  else.  In 
this  case,  L=n-2  and 

W  =  12S/n(n  —  2)2  (n2  —  1) 

The  data  matrix,  R,  corresponding  to  (1)  is  characterized  by  zero  on  the  major  diagonal.  The 
following  results  for  the  sampling  distribution  of  S  pertain  to  that  case. 

The  anticipated  outcomes  of  this  research  include: 

1 .  Increasing  our  understanding  of  the  dynamic  factors  driving  team  effectiveness 
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2.  Developing  theoretically  driven  and  empirically  based  guidelines  for  designing  and 
managing  effective  distributed  logistics  teams 

3.  Providing  a  research  platform  for  collaborative  logistics  tasks 

4.  Informing  the  design  and  implementation  of  collaborative  technologies  and  paradigms  for 
use  in  the  distributed  logistics  domain. 
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2.9  Appendixes 

2.9.1  Appendix  1:  Critical  Task  in  CAPS 


Note:  I:  Independent  task  that  is  not  necessary  to  cooperate  and  communicate  with  other  services;  D:  Dependent 
task  that  is  necessary  to  cooperate  and  communicate  with  other  services _ 


Functional 

Section 

Critical  Task 

Relation 

with 

other 

services 

Task  Description 

Air  Terminal 
Operations 

Flight  (ATOF): 

Provides  the  aerial 
port  sections  with 
information  to 
manage  available 
resources  to 
receive, 
document,  and 
move  passengers 
and  cargo. 

1.  Communicate 
Aircraft 
Requirements 

D 

•  When  an  aircraft  arrives  at  the  Air 
Terminal,  you  must  update  your  team 
members  with  information  about  the  aircraft 
so  that  they  can  plan  their  activities. 

•  You  must  inform  your  teammates  of  the 
following  information: 

■A  Destination  -  all  members 
•S  Location  -  all  members 
•S  Seats  available  -  Passenger  Services 
■A  Pallets  available  -  Cargo  Services  & 
Ramp  Services 

2.  Log  Flight 

Line  Activities 

D 

•  One  of  your  duties  is  to  keep  track  of  all 

activities. 

•  You’ll  need  to  record: 

■A  When  Passenger  Services  completes 
the  loading  and  unloading  of 
passengers. 

■A  When  Fleet  Services  completes  the 
servicing  of  an  aircraft  (cleaning  the 
aircraft  and  delivering  meals  to  the 
aircraft). 

■A  When  Ramp  Services  completes  the 
loading  and  unloading  of  cargo. 

3.  Port  Clear 

Aircraft 

I 

•  One  of  your  most  important  duties  is  to 
ensure  that  all  activities  have  been 
performed  on  an  aircraft  before  it  departs  - 
this  is  called  “port  clearing”  an  aircraft. 

•  After  you  clear  an  aircraft,  it  will  depart  the 
aerial  port  and  travel  to  its  next  destination. 

4.  Update  Team 
Members 

D 

•  You  are  the  information  manager  for  the 
entire  aerial  port.  You  need  to  keep  the 
various  services  (Fleet,  Passenger,  Cargo, 
and  Ramp)  up  to  date  with  the  latest 
information  as  you  receive  it  so  that  they 
can  perform  their  duties. 

•  You  will  also  need  to  reply  to  questions 
from  the  various  services. 
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Functional 

Section 

Critical  Task 

Relation 

with 

other 

services 

Task  Description 

Cargo  Services 
(CS):  In-processes 
in-bound  and  out- 

1 .  Unload 

Cargo 

Trucks 

D 

•  The  aerial  port  receives  cargo  from  trucks. 

•  This  cargo  must  be  taken  off  the  trucks  and 
moved  into  the  warehouse  for  processing. 
The  warehouse  stores  all  unprocessed  cargo. 

bound  cargo  and 
sequences 
palletized  cargo  for 
pick-up  by  ramp 
services. 

2.  In-process 
Cargo 

I 

•  Cargo  services  must  in-process  the  cargo  by 
assigning  pallets  to  bays. 

•  The  bays  should  be  organized  according  to 
the  destination  of  the  cargo.  For  example,  all 
cargo  going  to  Germany  may  be  stored  in  bay 

1. 

•  You  assign  cargo  to  specific  bays  so  that 
ramp  services  know  where  to  pick  up  the 
cargo. 

•  You  will  continually  in-process  cargo 
because  the  highest  priority  cargo  needs  to  be 
transported  first. 

3.  Sequence 

Bay  Cargo 

I 

•  Cargo  services  must  sequence  all  cargo 
within  a  bay  according  to  priority. 

•  The  system  will  automatically  sort  the  cargo 
for  you  when  you  tell  it  to. 

•  By  sequencing  the  cargo  in  a  bay,  you  make  it 
easy  for  ramp  services  to  go  to  a  bay  and 
select  the  appropriate  pallets. 

•  Remember  high  priority  pallets  (e.g.,  priority 
1)  must  be  shipped  before  low  priority  pallets 
(e.g.,  priority  2  or  3). 

•  Anytime  you  add  cargo  to  a  bay  you  must  re¬ 
sequence  the  cargo  in  the  bay  before  ramp 
services  can  take  any  pallets. 

4.  Notify 

ATOF  and 
Ramp 

Services 

After  Bay  Is 
Sequenced 

D 

•  You  need  to  IM  (instant  message)  Ramp 
Services  when  you  have  finished  sequencing 
a  cargo  load  so  that  they  can  pick  it  up  and 
deliver  it  to  the  aircraft.  Specifically,  they 
need  to  know  which  bay  the  load  is  in  and  to 
which  aircraft  it  should  be  delivered. 

•  Notify  ATOF  when  a  bay  has  been  sequenced 
by  opening  a  chat  window  via  the  Chat  menu. 

•  Type  your  message  in  the  bottom  text  box 
and  click  on  Send  button  or  press  Enter  key. 
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Functional 

Section 

Critical  Task 

Relation 

with 

other 

services 

Task  Description 

Fleet  Services 
(FS):  Supplies 
aircraft  with 
passenger  and 
crew  comfort  items 
(blankets,  pillows, 
meals,  etc.)  while 
ensuring  that 
transport  aircraft 
interiors  are  clean 

1 .  Clean 

Aircraft 

I 

•  One  of  your  duties  is  to  clean  the  inside  of 
the  aircraft  after  it  lands. 

•  You  can’t  clean  the  aircraft  until  both  the 
passengers  and  the  terminating  cargo  (i.e., 
the  cargo  that  is  staying  at  your  base)  are 
removed  from  the  plane. 

•  You  will  need  to  coordinate  your  activities 
with  passenger  services  and  ramp  services  to 
ensure  that  these  activities  are  done  before 
you  clean  the  aircraft. 

and  presentable. 

2.  Order  Flight 
Meals  and 
Restock 
Supplies  in 
FS  Truck 

D 

•  One  of  the  duties  of  Fleet  Services  is  to 
provide  aircraft  with  supplies  (such  as 
pillows,  blankets,  etc.) 

3.  Deliver 
Supplies 
(Including 
Flight 

Meals)  to 

Aircraft 

I 

•  One  of  the  duties  of  Fleet  Services  is  to 
deliver  meals  to  aircraft. 

•  You  will  need  to  know  how  many  passenger 
meals  to  order,  and  you  get  this  information 
from  Passenger  Services. 

•  The  meals  go  onto  the  plane  after  the 
passengers  are  boarded.  You  will  need  to 
coordinate  this  activity  with  Passenger 
Services  to  ensure  that  the  passengers  are  on 
the  plane  before  the  meals  are  loaded. 

4.  Notify 

ATOF  when 
each  activity 
is  completed 

D 

•  Y ou  need  to  keep  ATOF  updated  of  when  the 
aircraft  is  cleaned,  when  supplies  are 
delivered,  and  when  meals  are  delivered. 

•  After  you  complete  one  of  these  activities, 
you  should  send  ATOF  a  message  to  let  them 
know  that  that  task  is  complete. 

41 


Functional 

Section 

Critical  Task 

Relation 

with 

other 

services 

Task  Description 

Passenger  Services 
(PS):  In-processes 
passengers,  loads, 
and  unloads 
passengers  for 
aircraft  in  the  aerial 
port. 

1.  Unload 
(Disembark) 
Passengers 

I 

•  Aircraft  will  arrive  at  a  ramp  and  the 
passengers  on  board  need  to  be  unloaded. 

•  The  passengers  must  be  unloaded  before  any 
cargo  can  be  removed. 

•  You  will  move  a  bus  out  to  the  aircraft  to 
pick  up  the  passengers,  then  you  will  move 
the  bus  back  to  the  terminal  to  disembark  the 
passengers. 

•  To  “disembark”  the  passengers  means  to 
drop  them  off  at  the  terminal  and  means  they 
are  out  of  the  system. 

2.  In-Process 
Passengers 

I 

•  Before  passengers  can  fly  on  military  flights, 
they  must  be  entered  into  the  system. 

•  You  will  use  passengers’  social  security 
numbers  (SSN)  to  retrieve  information  about 
the  passenger. 

•  You  assign  passengers  to  a  gate  based  on 
their  desired  destination  (e.g.,  Germany). 

•  It  is  important  that  you  assign  all  passengers 
heading  to  the  same  destination  in  the  same 
gate. 

3.  Load 

Passengers 

D 

•  Before  an  aircraft  departs,  it’s  necessary  to 
load  the  passengers 

4.  Notify 

ATOF  of 

when 
passengers 
are  loaded  or 
unloaded 

D 

•  You  need  to  keep  ATOF  updated  of  when 
passengers  are  loaded  or  unloaded  so  that 
they  can  properly  keep  track  of  all  aerial  port 
activities. 

•  After  you  complete  a  task  (e.g.,  downloading 
or  uploading  passengers),  you  should  send 
ATOF  a  message  to  let  them  know  that  the 
task  is  complete. 
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Functional 

Section 

Critical  Task 

Relation 

with 

other 

services 

Task  Description 

Ramp  Services 
(RS):  Loads  out¬ 
bound  cargo  and 
downloads  in¬ 
bound  palletized 
cargo  for  the  aerial 
port. 

1 .  Unload 
Terminating 
Cargo  From 
Arriving 
Aircraft 

I 

•  When  aircraft  land,  they  often  have  cargo 
that  stays  at  your  home  base  which  is  Scott 
Air  Force  Base.  The  cargo  that  stays  at  Scott 
Air  Force  Base  is  referred  to  as  terminating 
cargo. 

•  Terminating  cargo  must  be  removed  to  make 
room  for  other  cargo  going  to  other 
destinations. 

•  Using  forklifts,  you  must  move  terminating 
cargo  from  the  aircraft  into  a  warehouse. 

•  The  forklifts  are  in  a  parking  lot  near  the 
Cargo  Services  building. 

2.  Load  Cargo 
To  Departing 
Aircraft 

D 

•  Cargo  needs  to  be  loaded  onto  departing 
aircraft. 

•  You  will  pick  up  the  cargo  from  bay  areas 
that  are  organized  by  cargo  services. 

•  Y ou  will  need  to  check  with  cargo  services  to 
see  where  the  cargo  is  (i.e.,  which  bay  #). 

•  Cargo  with  the  highest  priority  (e.g.,  priority 
1)  must  be  shipped  before  cargo  with  a  lower 
priority  (e.g.,  priority  2  or  3). 

•  Cargo  services  should  sequence  the  bays  so 
that  the  highest  priority  items  are  the  first  to 
be  picked  up. 

3.  Notify 

ATOF  When 
Loading  Or 
Unloading 

Are 

Completed 

D 

•  Y ou  need  to  keep  ATOF  updated  of  when  an 
aircraft  is  loaded  or  unloaded  so  that  they  can 
keep  track  of  the  information. 

•  After  you  either  load  or  unload  an  aircraft, 
you  should  send  ATOF  an  IM  (instant 
message)  to  let  them  know  that  the  task  is 
complete. 

43 


2.9.2  Appendix  2:  Interpersonal  Trust  (based  on  Naquin  &  Paulson,  2000) 


Scale:  (l=Strongly  Disagree,  2=Disagree,  3=Somewhat  Disagree,  4=Neither  Agree  nor  Disagree, 
5=Somewhat  Agree,  6=Agree,  7=Strongly  Agree) 

Instructions:  Using  the  scale  provided,  please  rate  your  agreement  with  the  following  items. 

■  I  think  other  team  members  told  the  truth  during  the  task 

■  I  think  other  team  members  met  their  obligations  during  the  task 

■  In  my  opinion,  other  team  members  are  reliable 

■  I  think  that  other  team  members  succeed  by  stepping  on  other  people  (-) 

■  I  feel  that  other  team  members  try  to  get  the  upper  hand  (-) 

■  I  think  other  team  members  took  advantage  of  my  problems  (-) 

■  I  feel  that  other  team  members  perfonned  the  task  with  me  honestly 

■  I  feel  that  other  team  members  will  keep  their  word 

■  I  think  other  team  members  have  misled  me  (-) 

■  I  feel  that  other  team  members  try  to  get  out  of  their  commitments  (-) 

■  I  feel  that  other  team  members  take  advantage  of  people  who  are  vulnerable  (-) 
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3 . 1  Executive  summary 

This  report  documents  a  research  project  focused  on  developing  a  protocol  for  assessing 
trust  in  virtual  teams.  Much  of  the  existing  literature  on  trust  in  virtual  teams  includes  case  study 
reports.  These  ethnographic  approaches  provide  rich  detail  of  work-team  interactions,  and 
provide  valuable  documentation  of  barriers  encountered  by  virtual  teams,  as  well  as  strategies 
that  seem  to  facilitate  teamwork  in  virtual  settings.  The  goal  of  this  effort  was  to  adapt  effective 
strategies  observed  in  natural  settings  for  use  in  a  controlled  experiment  that  would  allow  for 
manipulation  and  more  precise  examination  of  the  effectiveness  of  these  strategies.  Two  specific 
strategies,  tenned  “Break-the-Ice”  and  “Situation  Awareness  Calibration”  were  used  as 
interventions  in  this  study. 

Fourteen  three-member  teams  navigated  a  simulated  military  domain  (Virtual  Battlespace 
I)  to  complete  14  logistics  tasks.  Seven  teams  were  instructed  to  perform  the  interventions  and 
the  other  seven  teams  performed  the  tasks  without  being  instructed  to  do  so.  As  part  of  the 
scenario,  commanders  were  faced  with  a  choice  between  trusting  team  members  or  anonymous 
intel  data.  Following  the  completion  of  all  missions,  a  38  question  survey  (combination  of  5 
previously  validated  surveys)  was  administered  to  asses  the  trust  between  the  teammates. 

Perhaps  the  most  surprising  finding  was  that  participants  in  the  study  failed  to 
demonstrate  the  levels  of  distrust  reported  in  prior  research  (cites).  One  possible  explanation  for 
this  is  that  gamers  may  develop  strategies  to  assess  team  members  and  establish  trust  quickly. 
This  is  in  contrast  to  studies  of  trust  in  virtual  teams  in  business  settings  or  among  students 
working  on  group  projects. 

No  significant  difference  in  level  of  trust  was  observed  between  the  control  and 
experimental  groups.  Additional  research  is  needed  to  explore  whether  the  rapid  development  of 
trust  is  specific  to  the  gaming  community.  An  alternate  explanation  is  that  other  factors  such  as 
the  scenario,  duration  of  team  interaction,  etc.  simply  did  not  evoke  sufficient  engagement  to 
generate  feelings  of  distrust  reported  elsewhere. 

The  project  was  a  success  in  developing  a  protocol  for  studying  trust  in  virtual  teams  in  a 
controlled  environment.  In  spite  of  the  fact  that  no  significant  differences  were  found  between 
the  experimental  and  control  groups,  several  important  contributions  were  realized: 
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•  Variability  in  trust  scales.  Trust  measurement  scales  validated  in  different  settings 
evoked  reports  of  high  levels  of  trust,  with  the  exception  of  an  instrument  developed 
by  Jian,  Bisantz,  &  Drury  (2000).  The  Jian  et  al  survey  (2000)  evoked  reports  of 
distrust  in  team  members  such  as  deceptiveness,  suspicious  intent/action,  lack  of 
integrity,  and  unreliability.  Additional  research  is  needed  to  explore  differing  levels 
of  sensitivity  across  the  surveys. 

•  Protocol  for  studying  trust.  This  study  adapted  strategies  from  measuring  trust  from 
the  trust-in-automation  literature  and  the  interpersonal  trust  literature.  From  the  trust- 
in-automation  research  tradition,  we  incorporated  a  forced  choice  in  which  the 
commander  had  to  choose  whether  to  trust  his  teammates  or  an  anonymous  intel 
report.  In  addition,  we  adapted  surveys  used  to  assess  trust-in-automation  as  well  as 
interpersonal  trust. 

•  Balance  of  control.  As  with  any  laboratory  study,  the  design  of  this  experiment 
involved  the  weighing  of  a  number  of  tradeoffs.  A  highly-engaging  war  game  used  in 
order  to  engage  participants.  While  the  high-level  of  flexibility  allowed  for  an 
engaging  scenario,  it  also  allowed  participants  to  deviate  from  the  script  and  perform 
unrelated  actions.  Controlling  the  experiment  too  tightly  will  likely  distort  the  team 
dynamics  being  studied,  while  not  enough  control  makes  it  difficult  to  interpret 
findings. 

Introduction 

Many  sense  and  respond  logistics  (SRL)  tasks  are  completed  by  distributed  teams.  It  is 
important  to  identify  and  understand  the  human  aspects  of  collaborative  decision  making  to 
ensure  that  the  tasks  are  completed  successfully.  The  first  phase  of  this  research  identified  a  need 
for  more  research  in  the  area  of  trust  development  on  virtual  teams  (Hill,  Davis,  Vohra,  Militello, 
Bowman,  Bowers,  2006).  This  report  documents  a  study  designed  to  evaluate  the  effect  of  two 
possible  trust  facilitating  interventions  (Break-the-Ice,  and  Situation  Awareness  Calibration)  on 
trust  for  virtual  teams. 
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3.2  Objectives 

The  primary  objective  of  this  study  was  to  investigate  the  effects  of  the  two  previously 
mentioned  interventions  on  the  development  of  trust  between  teammates  performing  sense  and 
respond  logistics  tasks.  The  interventions  involved  the  teammates  sharing  biographical 
information  prior  to  beginning  the  logistics  tasks  (Break-the  Ice)  and  the  commander  perfonning 
the  situation  awareness  calibration  every  ten  minutes. 

3.3  Experimentation 

Prior  to  data  collection  at  Wright  State  University,  a  small  scale  pilot  study  was 
performed  in  collaboration  with  University  of  Dayton  Research  Institute  (Militello,  Bowers, 
Gripper,  2007).  The  methodology  utilized  for  this  pilot  study  was  slightly  modified  and  used  as  a 
framework  for  the  present  study.  The  facility  used  for  the  pilot  study  allowed  for  individual 
offices  for  each  player.  During  testing  at  WSU,  each  player  was  located  in  the  same  room.  The 
room  was  partitioned  off  to  avoid  players  viewing  their  teammate's  screens  and  the  players  were 
not  allowed  to  talk  to  each  other.  This  negated  the  need  for  cell  phones  and  reduced  the  physical 
workload  on  the  second  investigator. 

The  following  sections  outline  the  experimentation  used  for  data  collection  at  Wright 
State  University. 

3.3.1  Instrumentation 

The  software  applications  used  for  the  research  included  Virtual  Battlespace  One  (VBS 
1)  as  a  test  bed,  Platypus  to  distribute  surveys,  and  ScreenShot  Magic  version  3.0.  The  hardware 
consisted  of  three  OmniTech  ClientPro  PC  computers  equipped  with  Pentium  IV  processors  and 
Microsoft  Windows  XP  Professional,  and  three  17-inch  monitors  with  areas  set  to  1280  by  1024 
pixels.  The  input  devices  used  were  three  standard  keyboards  and  three  two-button  optical  wheel 
mice.  The  commander  of  the  experimental  group  was  provided  a  digital  desktop  clock  to  help 
him/her  determine  accurate  times  to  perform  the  situation  awareness  calibrations.  The  study  was 
conducted  in  the  AMOS  Laboratory  at  Wright  State  University. 
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3.3.2  Scenario 


There  were  no  changes  to  the  scenario  used  in  the  pilot  study  conducted  with  UDRI.  Its 
impetus  was  to  have  logistics  teams  of  three  people  complete  a  number  of  SRL  tasks  in  a  manner 
that  the  effects  of  the  potential  trust  facilitating  interventions  could  be  observed.  The  teams 
comprised  of  a  commander  and  two  soldiers  (alpha  and  bravo).  During  experimentation,  the 
commander  was  only  able  to  view  a  real  time  map  of  the  area  that  displayed  the  location  of 
friendly  units  and  possible  hostiles  (See  Figure  4  below).  Both  soldiers  were  only  able  to  view 
the  three  dimensional  environment  they  were  operating  in  (Figure  5  below).  The  soldiers  did  not 
have  access  to  the  real  time  map.  To  allow  for  scripting  some  of  the  interaction,  the  bravo  soldier 
was  played  by  a  confederate. 

The  location  of  the  scenario  was  the  imaginary  island  of  Al-Almar.  On  this  island,  the 
team  was  in  preparation  for  the  increase  of  coalition  forces  in  the  area.  Their  task  was  to 
inventory  equipment  available  in  different  locations  of  the  island. 


Figure  4.  Real  Time  Map  Available  to  Commander 
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Figure  5.  Example  Views  of  Soldiers 


At  specific  times  during  the  scenario,  the  commander  was  given  missions  that  needed  to 
be  disseminated  to  the  ground  soldiers.  Each  mission  included  orders  for  one  soldier  to  locate 
and  report  quantities  of  assets  at  a  particular  location.  Intelligence  data  was  provided  with  the 
missions  as  well  as  reminders  of  the  criticality  of  the  mission.  Some  of  the  intelligence  data  was 
designed  to  be  in  conflict  with  what  the  soldiers  would  be  able  to  account  for.  This  introduced 
circumstances  where  the  commander  would  have  to  decide  if  they  would  trust  the  intelligence 
reports  or  their  teammates.  The  commander  was  cautioned  of  the  inexperience  and  likelihood  of 
his  soldiers  to  get  confused  when  using  the  terrain  map  and  identifying  some  of  the  equipment.  It 
was  also  stressed  that  it  was  the  responsibility  of  the  commander  to  report  the  most  accurate 
information  available  whether  it  was  from  the  intelligence  reports  provided  with  the  missions  or 
the  responses  from  the  soldiers. 

3.3.3  Participants  and  Team  Development 

The  experimental  data  were  collected  from  28  Wright  State  University  students  ages  18  - 
31.  Of  the  28  participants,  13  were  native  English  speakers  and  15  speak  English  as  a  second 
language.  All  participants  were  required  to  have  experience  with  first-person  shooter  games  prior 
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to  their  participation  in  the  study.  An  online  screening  survey  developed  using  Platypus  was  used 
to  determine  experience.  We  were  unable  to  recruit  a  sufficient  number  of  expert  gamers  to 
complete  this  study.  Thus,  the  experience  level  of  the  player’s  was  variable,  but  was  balanced 
across  groups.  Of  the  28  players,  8  were  novice  to  computer  gaming,  13  had  intermediate  gaming 
experience,  and  7  were  experts.  Teams  of  two  were  formed  primarily  based  on  the  availability  of 
the  participants  and  the  experimenters.  Each  team’s  "method  of  play"  was  alternated  between 
experimental  and  control  groups  (i.e.,  the  first  team  was  a  control  group  and  the  second  team  was 
an  experimental  group).  There  were  only  2  teams  that  played  with  two  expert  players  (one  in  the 
control  group  and  one  on  the  experimental  group).  Participants  were  compensated  for  taking  part 
in  the  study  and  for  completing  all  missions  effectively  and  efficiently. 

3.3.4  Procedures 

The  basic  procedure  for  each  team  included  providing  consent  to  participate,  going 
through  a  training  exercise,  assignment  to  a  role  (commander  or  solider  based  on  performance 
during  training  exercise,  i.e.,  those  who  did  not  maneuver  well  were  assigned  to  commander 
role),  and  experimental  procedures.  The  following  sections  provide  more  details  on  the  training 
exercise  and  experimental  procedures.  Both  the  experimental  and  control  groups  received  the 
same  training. 


Training 

During  the  training  phase,  each  participant  was  given  time  to  familiarize  themselves  with 
the  game  and  to  learn  various  maneuvers  necessary  to  complete  the  scenario.  Game  play 
instructions  (Appendix  A)  and  a  key  map  (Appendix  B)  were  provided  during  the  training  and 
available  for  reference  throughout  the  testing  period. 

The  game  play  instructions  were  designed  to  walk  the  commander  and  alpha  soldier 
through  a  tutorial  that  demonstrated  the  following:  (1)  how  to  communicate  with  other 
teammates  using  the  chat  tool  in  VBS  1,  (2)  how  to  look  around  the  environment,  and  (3)  how  to 
perform  basic  movements  of  the  characters  and  vehicles.  The  tutorial  ended  with  the  participants 
getting  into  a  vehicle  and  using  the  map  to  get  to  the  airport.  Once  they  reached  the  airport  and 
exited  their  vehicle,  training  was  over  and  the  experimental  test  was  set  up.  The  key  map  was 
given  as  a  quick  reference  to  the  keyboard  functions. 
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Experimental  Procedures 

Following  the  training  session,  the  participants  were  told  that  the  game  would  take  a  few 
minutes  to  begin  and  to  wait  quietly.  During  this  time  the  commanders  of  the  experimental  teams 
were  given  the  situation  awareness  (SA)  calibration  sheet  (Appendix  C)  to  review.  This 
document  detailed  the  importance  of  the  calibration  and  instructs  the  participant  to  perform  the 
SA  calibration  approximately  every  10  minutes.  Also  during  this  waiting  period,  the  Bravo 
soldier  (e.g,  the  confederate)  of  the  experimental  group  began  the  “Break-the-Ice”  intervention 
by  initiating  an  infonnal  chat-based  in  exchange.  The  control  group  was  not  exposed  to  either 
intervention  and  sat  quietly  during  this  time. 

After  about  five  minutes  of  "game  set-up/intervention  set-up"  time,  the  commanders  and 
the  alpha  soldiers  of  both  the  experimental  and  control  groups  were  given  their  respective 
packets  of  information.  The  commander's  packet  included  and  information  sheet  (Appendix  D), 
resource  checklist  (Appendix  E),  and  an  example  mission  order  (Appendix  F).  The  information 
sheet  outlined  the  nature  of  the  mission,  role  of  the  commander,  information  about  the  team,  an 
explanation  of  mission  orders  and  mission  reports,  and  how  to  use  the  resource  checklist.  The 
commander's  information  sheet  also  relayed  the  inexperience  and  chances  of  confusion  of  his 
ground  soldiers.  The  experimenter  answered  any  questions  on  how  to  fill  out  the  resource 
checklist  and  the  contents  of  the  example  mission  order.  Once  the  commander  read  through  the 
all  of  the  packet  materials,  they  were  instructed  to  type  “I  am  your  new  commander.  I  will  be 
issuing  you  orders  from  now  on.”  This  communication  was  the  initiation  of  the  game. 

The  alpha  soldier  was  given  an  information  sheet  (Appendix  G)  that  outlined  the  nature 
and  goals  of  the  scenario,  the  role  of  the  alpha  soldier,  and  instructions  to  avoid  enemy  contact. 

In  addition,  the  alpha  soldier  was  given  a  US  Anny  reference  sheet  (Appendix  H)  and  an  enemy 
forces  reference  sheet  (Appendix  I)  for  quick  reference  of  friendly  and  enemy  forces  and 
equipment  during  game  play.  They  were  instructed  to  watch  their  screen  for  further  instructions 
from  the  commander.  Once  the  game  was  initiated,  the  commander  was  given  a  total  of  14 
mission  orders  at  previously  determined  times  (Appendix  J)  to  disseminate  to  his  teammates. 
After  all  missions  were  completed,  the  participants  were  asked  to  fill  out  the  trust  surveys  online. 
Following  the  surveys,  all  team  members  were  brought  together  and  debriefed. 
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3.4  Data  analysis 

Each  participant  in  the  commander  role  filled  out  the  resource  checklist.  There  were 
discrepancies  between  the  intelligence  reports  that  came  with  the  mission  orders  and  what 
resources  were  at  the  locations  for  four  of  the  missions.  For  example,  the  sixth  mission  for  the 
Bravo  soldier  was  to  count  tanks  at  Djebel  Gabr.  The  intelligence  report  alerts  the  commander 
that  there  are  4  tanks  there,  but  on  the  ground,  there  are  only  2.  The  commander  had  to  decide  if 
s/he  would  trust  the  intelligence  reports  or  their  teammates.  A  frequency  table  was  generated  for 
the  control  and  experimental  groups  based  on  the  responses  on  the  resource  checklist  (Table  14). 
It  was  considered  trust  in  the  teammate  if  the  commander  reported  anything  other  than  what  was 
on  the  intelligence  report.  From  the  data  provided,  the  commanders  from  both  the  groups  in 
general  reported  the  eyes  on  counts,  (or  trusted  their  teammates  more  than  the  intelligence 
reports).  A  chi-square  analysis  showed  no  significant  differences  between  the  groups  (%  ratio  = 
0.133,  p-value  =  0.72). 


Table  14.  Frequency  Table  for  Assessment  of  Commander's  Trust  of  Teammates 


Control  Teams 

Cumulative 

Trust 

Frequency 

Percent 

Frequency 

Cumulative  Percent 

Team  Mates 

24 

85.71 

24 

85.71 

Intelligence  Reports 

4 

14.29 

28 

100 

Experimental  Teams 

Team  Mates 

23 

82.14 

23 

82.14 

Intelligence  Reports 

5 

17.86 

28 

100 

Next,  t-tests  were  used  to  analyze  the  survey  assessments  for  the  teams.  The  survey  used  for 
team  trust  assessment  was  a  conglomerate  of  four  scales  that  have  been  validated  in  previous 
research.  The  questions  were  given  with  a  seven  point  likert  scale  response  mode  ranging  from 
strongly  agree  (1)  to  strongly  disagree  (7).  Table  15  shows  each  question,  and  its  source.  The 
statistical  data  analysis  was  performed  on  the  differences  between  the  responses  from  the 
experimental  groups  to  the  responses  from  the  control  groups.  These  results  are  discussed  in  the 
categories  described  in  the  source  and  purpose  column  of  Table  15.  In  addition,  to  the  questions 
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listed,  respondents  were  also  encouraged  to  record  their  impressions  of  the  session  and  examples 
of  events  that  influenced  their  trust  in  their  teammates.  These  results  are  discussed  as  well. 


Table  15.  Synopsis  of  Team  Trust  Assessment  Questionnaire 


Question 

Source  and  Purpose 

1 

If  I  had  my  way,  I  wouldn't  let  the  other  team  members  have  any 
influence  over  issues  that  are  important  to  the  team  mission 

“Measures  of  Trust” 

Jarvenpaa,  &  Leidner,  1999 

2 

I  would  be  comfortable  giving  the  other  team  members  complete 
responsibility  for  the  completion  of  the  mission 

3 

I  really  wish  I  had  a  good  way  to  oversee  the  work  of  the  other  team 
members 

*adapted  from  Mayer, 

Davis,  Shoorman,  1995 

4 

I  would  be  comfortable  giving  the  other  team  members  a  task  or  problem 
which  was  critical  to  the  mission,  even  if  I  could  not  monitor  them 

5 

Members  of  my  team  show  a  great  deal  of  integrity 

6 

I  can  rely  on  those  with  whom  I  work  on  this  team 

“Trust” 

7 

Overall,  the  people  on  my  team  are  very  trustworthy 

Jarvenpaa,  S.  &  Leidner,  D., 
1999 

8 

We  are  usually  considerate  of  one  another's  feelings  in  this  team 

9 

The  people  on  my  team  are  friendly 

*adapted  from  Pierce 

Sommer,  Morris,  Fridegar, 

10 

There  is  no  team  spirit  in  my  group 

11 

There  is  a  noticeable  lack  of  confidence  among  the  people  on  my  team 

1992 

12 

We  have  confidence  in  one  another  on  this  team 

13 

I  feel  like  I  belong  to  this  team 

14 

I  feel  like  I  am  a  member  of  this  team 

“Perceived  Team  Cohesion” 

15 

I  see  myself  as  part  of  this  team 

Bollen  &  Floyle,  1990 

16 

I  am  pleased  about  being  on  this  team 

17 

I  am  happy  to  be  a  part  of  this  team 

18 

Of  the  teams  I  have  played  with,  this  team  is  one  of  the  best 

19 

Members  of  this  team  are  deceptive 

20 

Members  of  this  team  behave  in  an  underhanded  manner 

21 

I  am  suspicious  of  team  members  intent,  actions,  or  outputs 

“Trust  in  Systems” 

22 

I  am  wary  of  other  team  members 

23 

A  team  members  actions  led  to  barm  fill  or  injurious  outcome  in  the  game 

Jian,  Bisantz,  Drary,  2000 

24 

I  have  confidence  in  the  members  of  this  team 

25 

The  members  of  this  team  provide  a  sense  of  security 

26 

This  members  of  this  team  have  integrity 
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27 

This  members  of  this  team  are  dependable 

28 

The  members  of  this  team  are  reliable 

29 

I  can  trust  the  members  of  this  team 

30 

I  am  familiar  with  the  members  of  this  team 

31 

Members  of  this  team  are  primarily  interested  in  their  own  welfare 

32 

There  are  times  when  members  of  this  team  cannot  be  trusted 

33 

Members  of  this  team  are  perfectly  honest  and  truthful  with  me 

“Interpersonal  Trust” 

34 

I  feel  that  I  can  trust  the  members  of  this  team  completely 

35 

The  members  of  this  team  are  truly  sincere  in  their  promises 

Larzelere  &  Huston,  1980 

36 

I  feel  that  members  of  this  team  do  not  show  me  enough  consideration 

37 

Members  of  this  team  treat  me  fairly  and  justly 

38 

I  feel  that  members  of  this  team  can  be  counted  on  to  help  me 

3. 4. 1  Statistical  Data  Analysis 

The  first  category  of  questions  relate  to  measures  of  trust.  Table  16  shows  the  mean  and  standard 
deviation  of  the  responses  for  the  commander,  alpha  solider  and  the  team  for  the  control  and 
experimental  groups.  To  compare  the  differences  between  the  groups,  t-tests  were  performed  on 
the  experimental  and  control  groups.  The  columns  labeled  “p-value”  show  the  results  for  each 
question  for  the  scale.  There  were  no  statistical  differences  between  the  responses  for  the  control 
and  experimental  groups.  Overall,  the  participants  were  not  comfortable  giving  influence  or 
responsibility  to  their  teammates.  In  terms  of  overseeing  the  team,  it  was  felt  that  the  activity  of 
overseeing  was  important  but  not  a  necessity  even  for  critical  conditions. 
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Table  16.  Descriptive  and  Statistical  Results  for  "Measures  of  Trust"  Scale 


Question 

Commander 

Alpha 

Team 

Mean 

(std.  dev.)  p-value 

Mean 

(std.  dev.)  p-value 

Mean 

(std.  dev.)  p-value 

Con.  Exp. 

Con.  Exp. 

Con.  Exp. 

1 )  If  I  had  my  way,  I 
wouldn't  let  the  other 
team  members  have 
any  influence  over 
issues  that  are 
important  to  the  team 
mission 

1.86  1.71 

(1.21)  (1.11)  0.82 

2.00  2.00 
(1.15)  (1.41)  1.00 

1.93  1.86 

(1.14)  (1.23)  0.87 

2)  I  would  be 
comfortable  giving 
the  other  team 
members  complete 
responsibility  for  the 
completion  of  the 
mission 

4.29  5.71 

(2.43)  (1.98)  0.25 

6.57  5.71 

(1.13)  (2.36)  0.40 

5.43  5.71 

(2.17)  (2.09)  0.73 

3)  I  really  wish  I  had 
a  good  way  to 
oversee  the  work  of 
the  other  team 
members 

2.43  1.86 

(1.90)  (1.86)  0.58 

2.00  1.57 

(1.15)  (1.13)  0.50 

2.21  1.71 

(1.53)  (1.49)  0.39 

4)  I  would  be 
comfortable  giving 
the  other  team 
members  a  task  or 
problem  which  was 
critical  to  the 
mission,  even  if  I 
could  not  monitor 
them 

1.29  1.14 

(0.49)  (0.38)  0.55 

2.14  1.57 

(1.35)  (1.13)  0.41 

1.71  1.36 

(1.07)  (0.84)  0.34 

The  second  scale  was  developed  to  assess  the  perceived  trustworthiness  of  the  team  members. 
Table  17  provides  the  mean,  standard  deviations,  and  the  t-test  results  of  the  responses.  There 
were  no  statistically  significant  differences  between  the  groups  for  this  scale.  The  respondents 
recorded  relatively  high  levels  of  integrity,  reliability,  and  trustworthiness  as  their  perceptions  of 
their  teammates.  They  also  felt  that  their  teammates  were  considerate  of  each  other,  but  not 
friendly  and  showed  fairly  low  team  spirit.  Another  interesting  find  from  this  scale  was  that  there 
was  a  noticeable  lack  of  confidence  in  the  actions  of  the  teammates,  but  an  overall  confidence  in 
each  other. 
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Table  17.  Descriptive  and  Statistical  Results  for  "Trustworthiness"  Scale 


Commander 

Alpha 

Team 

Mean 
(std.  dev.) 

p-value 

Mean 
(std.  dev.) 

p-value 

Mean 
(std.  dev.) 

p-value 

Question 

Con. 

Exp. 

Con. 

Exp. 

Con. 

Exp. 

5)  Members  of  my 
team  show  a  great 
deal  of  integrity 

1.86 

(1.46) 

1.57 

(0.79) 

0.66 

2.00 

(1.15) 

2.43 

(1.62) 

0.58 

1.93 

(1.27) 

2.00 

(1.30) 

0.88 

6)  I  can  rely  on 
those  with  whom  I 
work  on  this  team 

1.86 

(0.90) 

1.71 

(1.50 

0.83 

1.86 

(1.21) 

2.00 

(1.41) 

0.84 

1.86 

(1.03) 

1.86 

(1.41) 

1.00 

7)  Overall,  the 
people  on  my  team 
are  very 
trustworthy 

1.43 

(0.79) 

1.86 

(1.86) 

0.59 

1.71 

(0.95) 

1.71 

(1.50) 

1.00 

1.57 

(0.85) 

1.79 

(1.63) 

0.67 

8)  We  are  usually 
considerate  of  one 
another's  feelings 
in  this  team 

1.71 

(0.95) 

1.43 

(0.53) 

0.51 

2.14 

(1.46) 

2.14 

(1.35) 

1.00 

1.93 

(1.21) 

1.79 

(1.05) 

0.74 

9)  The  people  on 
my  team  are 
friendly 

6.71 

(0.49) 

5.43 

(2.07) 

0.16 

5.00 

(2.38) 

5.00 

(2.00) 

1.00 

5.86 

(1.88) 

5.21 

(1.97) 

0.38 

10)  There  is  no 
team  spirit  in  my 
group 

2.00 

(0.82) 

1.57 

(1.13) 

0.43 

2.57 

(1.27) 

2.00 

(1.41) 

0.44 

2.29 

(1.07) 

1.79 

(1.25) 

0.27 

1 1 )  There  is  a 
noticeable  lack  of 
confidence  among 
the  people  on  my 
team 

1.43 

(0.79) 

2.00 

(1.83 

0.47 

1.29 

(0.76) 

1.86 

(1.21) 

0.32 

1.36 

(0.74) 

1.93 

(1.49) 

0.22 

12)  We  have 
confidence  in  one 
another  on  this 
team 

1.00 

(1.14) 

2.00 

(2.24) 

0.35 

2.14 

(2.27) 

1.29 

(0.76) 

0.37 

1.64 

(1.65) 

1.64 

(1.65) 

1.00 

The  results  from  the  “Perceived  Cohesiveness”  survey  are  listed  in  Table  18.  Again,  there  were 
no  statistically  significant  differences  between  the  experimental  and  control  groups.  In  general, 
high  levels  of  cohesion  were  observed  on  the  teams,  both  experimental  and  control,  even  though 
the  sense  of  belonging  was  somewhat  lower  than  the  other  measures  of  cohesion. 
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Table  18.  Descriptive  and  Statistical  Results  for  "Perceived  Cohesion"  Scale 


Commander 

Alpha 

Team 

Mean 
(std.  dev.) 

p-value 

Mean 
(std.  dev.) 

p-value 

Mean 
(std.  dev.) 

p-value 

Question 

Con. 

Exp. 

Con. 

Exp. 

Con. 

Exp. 

13)1  feel  like  I 
belong  to  this  team 

5.43 

(2.30) 

6.00 

(1.83) 

0.62 

4.86 

(2.48) 

5.71 

(1.80) 

0.47 

5.14 

(2.32) 

5.86 

(1.75) 

0.37 

14)  I  feel  like  I  am 
a  member  of  this 
team 

1.71 

(1.11) 

1.57 

(0.79) 

0.79 

2.57 

(1.72) 

2.29 

(1.60) 

0.75 

2.14 

(1.46) 

1.93 

(1.27) 

0.68 

15)1  see  myself  as 
part  of  this  team 

1.57 

(0.79) 

1.57 

(0.79) 

1.00 

1.86 

(1.21) 

1.86 

(1.07) 

1.00 

1.71 

(0.99) 

1.71 

(0.91) 

1.00 

16)  I  am  pleased 
about  being  on  this 
team 

1.43 

(0.79) 

2.14 

(1.77) 

0.36 

1.57 

(0.98) 

1.71 

(1.11) 

0.80 

1.50 

(0.85) 

1.93 

(1.44) 

0.35 

17)  I  am  happy  to 
be  a  part  of  this 
team 

1.57 

(0.53) 

1.29 

(0.49) 

0.32 

2.00 

(1.91) 

1.86 

(1.21) 

0.87 

1.79 

(1.37) 

1.57 

(0.94) 

0.63 

18)  Of  the  teams  I 
have  played  with, 
this  team  is  one  of 
the  best 

2.14 

(1.46) 

2.86 

(2.54) 

0.53 

2.00 

(1.29) 

1.71 

(1.11) 

0.67 

2.07 

(1.33) 

2.29 

(1.98) 

0.74 

The  “Trust  in  Systems”  scale  was  created  to  evaluate  how  much  trust  a  person  has  in  a  system  or 
automation.  The  questions  have  been  slightly  modified  to  assess  trust  in  virtual  teams.  The  mean, 
standard  deviation,  and  t-test  results  are  presented  in  Table  19  below.  There  were  no  statistically 
significant  differences  between  groups  here  as  well.  The  participants  here  felt  that  their 
teammates  were  deceptive,  and  had  suspicious  intent  or  actions,  but  they  did  have  confidence  in 
each  other.  Here  it  was  also  recorded  that  the  team  had  little  integrity,  was  unreliable,  and 
unfamiliar  with  each  other.  The  responses  were  almost  neutral  on  the  subjects  of  team  security 
and  team  trust. 
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Table  19.  Descriptive  and  Statistical  Results  for  "Trust  in  Systems"  Scale 


Commander 

Alpha 

Team 

Mean 
(std.  dev.) 

p-value 

Mean 
(std.  dev.) 

p-value 

Mean 
(std.  dev.) 

p-value 

Question 

Con. 

Exp. 

Con. 

Exp. 

Con. 

Exp. 

19)  Members  of 
this  team  are 
deceptive 

1.57 

(1.13) 

2.57 

(2.07) 

0.29 

1.29 

(0.49) 

2.14 

(1.35) 

0.15 

1.43 

(0.85) 

2.36 

(1.63) 

0.08 

20)  Members  of 
this  team  behave  in 
an  underhanded 

manner 

2.86 

(2.48) 

5.00 

(2.77) 

0.43 

4.71 

(1.11) 

5.14 

0.66 

4.29 

(2.20) 

5.07 

(2.16) 

0.35 

21)  I  am 

suspicious  of  team 
members  intent, 
actions,  or  outputs 

1.71 

(0.95) 

1.57 

(0.79) 

0.76 

1.71 

(1.11) 

1.71 

(1.50) 

1.00 

1.71 

(0.99) 

1.64 

(1.15) 

0.86 

22)  I  am  wary  of 
other  team 
members 

1.57 

(1.13) 

1.86 

(1.86) 

0.74 

1.43 

(0.79) 

2.14 

(1.35) 

0.25 

1.50 

(0.94) 

2.00 

(1.57) 

0.32 

23)  A  team 
members  actions 
led  to  harmful  or 
injurious  outcome 
in  the  game 

1.57 

(0.79) 

1.43 

(0.53) 

0.70 

1.57 

(1.13) 

1.86 

(1.86) 

0.74 

1.57 

(0.94) 

1.64 

(1.34) 

0.87 

24)  I  have 
confidence  in  the 
members  of  this 
team 

2.71 

(2.21) 

3.00 

(1.73) 

0.79 

3.14 

(0.90) 

2.43 

(1.51) 

0.31 

2.93 

(1.64) 

2.71 

(1.59) 

0.73 

25)  The  members 
of  this  team 
provide  a  sense  of 
security 

3.00 

(2.08) 

2.14 

(1.07) 

0.36 

3.57 

(1.62) 

2.86 

(2.27) 

0.51 

3.29 

(1.82) 

2.50 

(1.74) 

0.25 

26)  This  members 
of  this  team  have 
integrity 

4.43 

(2.51) 

6.14 

(1.86) 

0.17 

5.86 

(1.68) 

5.57 

(2.15) 

0.79 

5.14 

(2.18) 

5.86 

(1.96) 

0.37 

27)  This  members 
of  this  team  are 
dependable 

3.14 

(1.95) 

3.00 

(2.08) 

0.90 

2.71 

(2.06) 

2.43 

(2.30) 

0.81 

2.93 

(1.94) 

2.71 

(2.13) 

0.78 
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28)  The  members 
of  this  team  are 
reliable 

6.43 

(1.13) 

5.86 

(1.86) 

0.50 

5.43 

(2.70) 

5.43 

(2.44) 

1.00 

5.93 

(2.06) 

5.64 

(2.10) 

0.72 

29)  I  can  trust  the 
members  of  this 
team 

2.14 

(1.86) 

3.14 

(1.95) 

0.35 

2.57 

(1.72) 

2.57 

(1.51) 

1.00 

2.36 

(1.74) 

2.86 

(1.70) 

0.45 

30)  I  am  familiar 
with  the  members 
of  this  team 

5.00 

(2.00) 

5.71 

(1.98) 

0.51 

6.29 

(1.25) 

4.71 

(1.80) 

0.09 

5.64 

(1.74) 

5.21 

(1.89) 

0.54 

The  final  scale  used  was  developed  to  determine  “Interpersonal”  trust.  Table  20  shows  the 
descriptive  statistics  and  the  t-test  results.  None  of  the  differences  in  the  responses  between  the 
experimental  and  control  groups  were  significant.  The  overall  responses  for  these  measures 
showed  positive  ratings  of  interpersonal  trust.  Players  did  not  observe  selfish  behaviors  during 
play  and  felt  that  their  teammates  were  trustable,  honest,  considerate,  fair  and  reliable. 
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Table  20.  Descriptive  and  Statistical  Results  for  "Interpersonal  Trust"  Scale 


Commander 

Alpha 

Team 

Mean 
(std.  dev.) 

p-value 

Mean 
(std.  dev.) 

p-value 

Mean 
(std.  dev.) 

p-value 

Question 

Con. 

Exp. 

Con. 

Exp. 

Con. 

Exp. 

31)  Members  of 
this  team  are 
primarily 
interested  in  their 
own  welfare 

5.71 

(2.21) 

5.57 

(1.99) 

0.90 

6.29 

(1.89) 

6.29 

(1.89) 

1.00 

6.00 

(2.00) 

5.93 

(1.90) 

0.92 

32)  There  are 
times  when 
members  of  this 
team  cannot  be 
trusted 

6.43 

(1.13) 

5.86 

(1.86) 

0.50 

5.57 

(2.15) 

6.29 

(1.50) 

0.49 

6.00 

(1.71) 

6.07 

(1.64) 

0.91 

33)  Members  of 
this  team  are 
perfectly  honest 
and  truthful  with 

me 

2.00 

(1.00) 

1.71 

(1.11) 

0.62 

3.86 

(2.12) 

3.00 

(2.00) 

0.45 

2.93 

(1.86) 

2.36 

(1.69) 

0.40 

34)  I  feel  that  I  can 
trust  the  members 
of  this  team 
completely 

2.00 

(1.41) 

1.57 

(0.79) 

0.50 

1.86 

(1.07) 

2.86 

(1.86) 

0.25 

1.93 

(1.21) 

2.21 

(1.53) 

0.59 

35)  The  members 
of  this  team  are 
truly  sincere  in 
their  promises 

0.49 

(1.57) 

6.29 

(0.95) 

0.07 

6.14 

(1.21) 

5.86 

(1.68) 

0.72 

5.50 

(1.51) 

6.07 

(1.33) 

0.30 

36)  I  feel  that 
members  of  this 
team  do  not  show 
me  enough 
consideration 

4.29 

(1.98) 

6.00 

(1.83) 

0.12 

5.71 

(2.21) 

6.00 

(1.00) 

0.76 

5.00 

(2.15) 

6.00 

(1.41) 

0.16 

37)  Members  of 
this  team  treat  me 
fairly  and  justly 

1.29 

(0.49) 

1.14 

(0.38) 

0.55 

2.14 

(1.57) 

1.57 

(0.79) 

0.41 

1.71 

(1.20) 

1.36 

(0.63) 

0.34 

38)  I  feel  that 
members  of  this 
team  can  be 
counted  on  to  help 
me 

1.43 

(0.79) 

1.71 

(1.15) 

0.67 

2.29 

(1.89) 

1.43 

(1.13) 

0.84 

1.86 

(1.46) 

1.57 

(1.28) 

0.59 
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3. 4.2  Qualitative  Analysis 


Along  with  the  likert  scale  questions  discussed  in  the  previous  section,  participants  were 
asked  to  provide  specific  examples  of  events  or  actions  that  influenced  their  trust  in  their 
teammates.  Table  21  summarizes  the  responses  for  this  question.  In  the  table,  each  row 
represents  a  team.  The  responses  here  point  to  a  relationship  between  the  quality  of  the 
communication  and  trust  rather  than  the  quantity  of  the  communication  and  trust.  This  was 
especially  evident  with  the  commanders  of  the  experimental  teams  2,  6  and  10  and  control 
commanders  of  teams  5  and  7.  It  was  also  observed  from  these  teams’  commanders  (2,  6,  and  10) 
that  when  the  soldiers  took  their  time  to  complete  missions,  it  had  a  positive  effect  on  their  trust 
of  the  soldiers.  Another  trend  drawn  from  these  responses  was  the  relationship  between  trust  and 
the  teammate’s  readiness  to  help  other  members  of  the  team.  Most  of  the  alpha  soldiers  stated 
actions  and  events  regarding  either  the  commander  or  the  bravo  soldier  helping  them  to  navigate 
the  game  or  complete  missions  as  events  that  influenced  their  trust.  Negative  comments  are  also 
listed  in  Table  21  that  point  to  the  helpfulness  of  team  mates  being  positively  correlated  with 
trust. 


Table  21.  Commander  and  Alpha  Soldier's  Examples  of  Events  or  Actions  that  Influenced  Their 

Trust  in  Their  Teammates 


# 

Control  Commander’s 

Control  Alpha’s 

1 

“I  would  perform  better  when  I  do  the  role  a  second  time;  more 
time  is  required  for  me  to  get  a  hang  of  the  game”. 

“When  near  the  end,  I  was 
confused  with  the  map  and 
everyone  was  helping  me  to 
figure  out  where  to  go,  I  felt  like 
I'm  part  of  the  team”. 

3 

“Bravo  was  on  par  and  fast.  He  stuck  with  all  his  missions,  and 
had  success.  Strong  trust  in  him  but  unfortunately  he  found  a 
few  AI  in  the  desert  and  was  terminated  repeatedly  which 
removed  him  from  the  game.  Alpha  missed  the  3rd  mission 
order  which  I  typed  at  least  twice  and  therefore  we  had  to  do  it 
at  the  end  of  the  game.  He  had  a  few  issues  with  remembering 
orders.  Trust  with  him  was  lower  than  with  Bravo.” 

“Bravo  was  sent  to  assist  me  in 
times  of  trouble”. 

5 

“When  I  had  told  them  earlier  to  report  in  when  they  made  it  to 
a  waypoint  and  then  they  did,  that  let  me  know  that  they  were 
serious  about  getting  the  mission  done  and  that  I  could  trust 
them  to  do  their  job.” 

“The  commander  was  quick  to 
give  me  my  orders  once  I 
reached  a  destination,  which  let 
me  trust  that  he  knew  what  he 
was  doing”. 

7 

“I  trusted  both  of  my  team  members.  When  the  Intel  they 

“I  did  not  see  collective  team 
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gathered  was  different  from  the  Intel  gathered  by  the  satellite,  I 
had  them  double  check.  But  other  than  that,  I  trusted  them”. 

work;  we  just  followed  the 
individual  orders  respectively 
though  I  didn’t  complete  them”. 

9 

“When  Alpha  was  in  trouble  bravo  went  to  pick  him  up  despite 
of  where  he  was.  Both  of  them  worked  as  a  team”. 

“When  I  asked  for  help,  the  team 
responded”. 

Experimental  Commander’s 

Experimental  Alpha’s 

2 

“Both  teams  were  responsive  and  clear,  and  this  made  me  trust 
both,  they  also  seemed  to  take  the  necessary  time  to  complete  a 
full  count,  and  so  I  went  with  their  reports”. 

“While  pinned  down  under 
enemy  fire,  the  commander  did 
nothing  to  aid  me.  Instead,  Bravo 
traveled  to  my  aid”. 

4 

No  Response 

“Bravo  came  to  pick  me  up  when 

I  got  stuck”. 

6 

“Well,  Bravo  was  never  injured,  so  that  made  me  trust  him  a  bit 
more.  Also,  he  seemed  more  likely  to  be  able  to  give  counts 
quickly  and  accurately.  Alpha  was  more  detailed  orientated, 
which  was  nice,  but  cost  some  delays  at  time”. 

“Bravo  drove  out  to  pick  me  up 
when  I  was  injured”. 

8 

“When  Alpha  was  in  trouble  and  not  able  to  get  out  from  one 
site  due  to  enemy  firing,  Bravo  was  ready  to  help  him  out.  In 
fact,  on  many  occasions  I  interchanged  their  tasks  so  as  to  get 
the  mission  done  as  early  as  possible”. 

“When  my  hummer  broke  down, 
Bravo  helped  me”. 

10 

“Confirmation  after  completing  missions  increases  my  trust  as 
well  as  time  spent  performing  the  mission  successfully.  My 
team  was  evenly  effective”. 

“The  commander  did  not  always 
respond  to  my  reporting,  so  I  was 
unsure  if  he  was  receiving  my 
orders.  As  a  result,  I  constantly 
repeated  myself’. 

3.5  Discussions  and  conclusions 

This  project  was  a  success  in  developing  a  new  paradigm  for  studying  trust  development 
in  virtual  teams.  Although  no  significant  differences  were  found  between  the  control  and 
experimental  groups,  there  were  a  number  of  contributions  and  significant  findings.  Some  major 
contributions  of  the  research  include: 

■  Trust  measurement  scales  from  different  domains  can  be  used  for  evaluating  virtual  team 
trust. 

■  A  good  benchmark  protocol  for  the  study  of  trust  in  virtual  teams  was  developed  and 
ways  to  refine  it  were  observed  during  implementation  of  this  study. 

■  Balance  of  control  in  both  the  scenario  and  instructions  is  important  to  the  reliability  of 
results. 

Subjective  surveying  is  the  most  common  method  of  measuring  trust.  There  are  a  number  of 
scales  that  have  been  developed  for  various  environments  and  relationships  including  trust  in 
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systems,  team  cohesion,  and  interpersonal  trust.  This  research  has  shown  that  these  scales, 
though  developed  and  used  in  other  domains,  can  be  successfully  used  in  evaluating  trust  in 
virtual  teams.  The  scale  developed  for  trust  in  systems  (Jian,  Bisantz,  Drury,  2000)  was  the  only 
one  that  seemed  to  have  evoked  slightly  different  reactions  from  respondents.  The  questions 
from  this  scale  were  developed  empirically  and  were  more  direct  in  nature.  These  types  of 
questions  may  provide  a  better  look  into  the  constructs  of  trust  to  provide  a  more  in  depth 
analysis  of  trust  in  virtual  teams.  It  was  demonstrated  here  that  a  combination  of  types  of  scales 
can  be  advantageous  in  looking  at  the  multifaceted  concept  of  trust. 

Trust  in  virtual  teams  is  a  very  complex  area  of  research  that  requires  a  very  detailed 
protocol,  especially  in  a  gaming  environment.  As  the  protocol  for  this  study  was  developed  and 
executed,  some  important  requirements  were  determined.  First,  without  face-to-face  interaction, 
we  anticipate  that  we  will  need  to  develop  longer  scenarios  with  more  interactions  between  team 
members  to  observe  distrust  and  trust  development  in  ad  hoc  virtual  teams  as  reported  elsewhere 
in  the  literature.  Also,  our  participants  were  compensated  for  their  time  but  seemed  not  invested 
in  the  task.  Gamers  can  be  considered  more  competitive  in  nature  and  if  using  this  type  of 
platform  and  participants,  the  stakes  should  be  higher  to  produce  more  personal  investments  in 
the  tasks  along  with  the  final  results.  Thus  a  protocol  for  using  games  as  a  test  bed  for  studying 
trust  in  virtual  teams  should  include  a  balanced  combination  of  high  stakes  and  necessities  to  rely 
on  teammates. 

It  was  also  observed  that  a  balance  of  control  is  a  definite  necessity.  For  this  type  of 
research,  there  is  a  need  for  a  mix  of  real  world  interactions  with  some  control.  This  control  is 
what  makes  this  a  complex  area  of  research,  but  the  delicacy  of  it  can  make  it  even  more  difficult 
obtain  accurate  results.  Too  little  or  too  much  control  can  have  major  implications  on  the 
findings.  As  the  data  collection  procedures  progressed,  the  importance  of  the  balance  of  control 
and  real  world  interaction  became  more  evident,  especially  in  terms  of  instructions  given  and  the 
scenario.  Both  instructions  and  the  scenario  have  influence  on  the  real  world  aspects  of  the 
experience.  Too  much  control  in  the  scenario  will  compromise  the  real  world  characteristics  of 
the  process  and  too  much  or  too  little  control  in  the  instructions  can  lead  to  players  getting  off 
course.  The  balance  of  control  for  these  two  must  be  taken  into  consideration  to  get  reliable 
interactions  and  results. 
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In  summary  of  the  research  findings,  a  chi-square  analysis  on  the  responses  from  the 
resource  checklists  showed  no  significant  differences  in  the  control  and  experimental 
commander’s  trust  of  their  teammates  (p-value  =  0.72).  During  testing,  it  was  observed  that  some 
of  the  control  commanders  performed  the  SA  calibration  by  instinct.  It  is  possible  that  this  is  one 
reason  why  there  are  no  differences  between  the  groups.  Both  groups  trusted  their  teammates 
over  the  intelligence  reports  more  than  80%  of  the  time  (85.71%  for  the  control  group  and  82.14 
for  the  experimental  group). 

The  survey  questions  were  analyzed  using  t-tests.  Again,  the  differences  between  the 
control  and  experimental  participants  were  not  significant  although  there  were  a  number  of 
interesting  findings  from  the  survey.  There  was  a  relatively  high  level  of  trust  for  both  groups. 

The  first  set  of  questions  was  pulled  from  a  survey  from  (Jarvenpaa,  &  Leidner,  1999). 
The  responses  here  showed  that  the  players  were  not  completely  comfortable  giving  influence  or 
responsibility  to  their  teammates  and  felt  that  overseeing  was  important  but  not  necessary.  There 
seems  to  be  trust  in  the  teams  but  with  hesitation. 

The  next  group  of  questions  looked  at  perceived  trustworthiness  of  the  teams.  The 
respondents  reported  high  levels  of  integrity,  reliability  and  trustworthiness  in  their  teammates. 
There  was  also  a  displayed  confidence  in  the  overall  teams,  but  the  teammates  noticed  a  lack  of 
confidence  among  the  individuals  on  the  team.  In  other  words,  even  though  individuals  on  the 
teams  did  not  show  a  lot  of  confidence  in  themselves  and  their  abilities,  the  overall  view  of  the 
team  was  not  affected  by  it. 

In  terms  of  team  cohesion,  both  the  control  and  experimental  team  members  felt  unified  and 
somewhat  pleased  to  be  on  the  team.  The  sense  of  belonging  was  not  readily  observed  in  the 
survey  responses  for  either  of  the  groups.  An  alpha  solider  stated  that  they  did  not  see  collective 
teamwork  and  that  they  just  followed  individual  orders.  This  could  explain  some  of  the  lack  of 
sense  of  belonging  to  the  team.  Some  participants  may  have  felt  that  they  were  on  a  team  in  a 
sense  that  they  were  working  together  to  complete  the  missions  but  not  necessarily  interrelated  to 
the  team.  There  was  a  question  on  the  previous  scale  about  team  spirit.  There  was  some 
agreement  among  teams  that  there  was  no  team  spirit. 

The  next  scale  was  the  modified  “trust  in  systems”  scale.  There  responses  were  somewhat 
conflicting  with  those  in  some  of  the  other  scales.  Both  groups  reported  levels  of  deception, 
suspicious  intent  and  actions,  little  integrity,  little  reliability  and  little  familiarity  with  each  other. 
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The  final  scale  measured  interpersonal  trust.  The  ratings  here  were  relatively  high  for 
interpersonal  trust.  Overall,  both  groups  felt  that  their  teammates  were,  trustable,  honest, 
considerate,  fair  and  reliable. 

When  asked  to  provide  examples  of  events  or  actions  that  influenced  their  trust  in  their 
teammates,  the  players  listed  comments  that  led  to  the  possible  relationship  between  the  qualities 
of  communication  over  the  quantity.  An  alpha  solider  was  criticized  for  being  more  detailed 
oriented  (Experimental  Team  6)  which  caused  time  delays,  whereas  the  bravo  solider  of  the 
same  team  responded  quickly  and  accurately  was  trusted  more.  Also,  the  alpha  soldiers  tended  to 
relate  assistance  from  teammates  with  their  trust  of  them.  Commanders  on  the  other  hand, 
seemed  to  relate  trust  to  how  long  it  took  to  complete  missions.  When  soldiers  took  their  time  to 
complete  missions,  commanders  felt  like  they  were  more  “committed”  to  the  missions. 

Another  major  observation  from  this  study  was  that  though  the  trust  ratings  were 
relatively  high,  these  particular  interventions  seemingly  had  little  effect  on  the  development  of 
trust  on  the  teams.  High  levels  of  trust  reported  by  participants  in  this  study  are  surprising  given 
findings  from  studies  of  trust  in  virtual  teams  in  the  workplace.  These  prior  studies  suggest  that 
trust  is  generally  slow  to  develop  and  particularly  fragile  when  team  members  do  not  get  an 
opportunity  to  interact  face-to-face  (Burgoon  et.  al.,  2003;  Jarvenpaa  &  Leidner,  1999).  Wainfan 
and  Davis  (2004)  summarize  their  extensive  review  of  the  literature  with  these  words:  “[in 
virtual  teams]  participation  is  reduced,  along  with  interpersonal  factors  such  as  perceptions  of 
other  collaborators,  cohesiveness,  persuasiveness,  cooperation,  and  leadership  emergence. 

Virtual  groups  are  more  likely  to  attribute  others’  behavior  less  generously,  to  dispositional 
rather  than  situational  factors.”  (p.  61).  Our  study  did  not  support  this  previous  finding.  We 
speculate  that  in  the  gaming  culture,  participants  tend  to  be  more  strategically  inclined.  The 
strategies  employed  here  seemed  to  be  conducive  to  building  trust  in  teammates  quickly. 

In  terms  of  the  game,  there  were  several  bugs  that  were  discovered  during  play.  The 
players  could  only  view  3  lines  of  the  chat  log  at  a  time  and  could  not  scroll  up  if  they  missed 
something.  If  one  player  was  engaged  in  conversation  with  another  and  the  third  team  member 
responded  to  a  mission,  it  was  often  missed  (the  third  team  member  had  no  way  of  knowing  if 
their  response  was  recorded  unless  the  commander  somehow  acknowledged  it)  and  the  third 
player  would  have  to  repeat  themselves.  This  could  be  addressed  by  having  some  type  of 
transcribing  mechanism  in  a  separate  window  for  the  commanders  to  scroll  through.  Also,  there 
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were  some  complaints  on  the  sensitivity  of  the  mouse  as  related  to  controlling  vehicles  and 
issues  of  skill  needed  when  driving  the  vehicles. 

3.6  Recommendations  for  future  work 

There  is  a  need  for  further  research  into  trust  development  on  virtual  teams.  This  study  is 
a  good  benchmark  for  potential  future  work  investigating  issues  of  trust  in  virtual  teams  using 
computer  based  games.  Here,  we  looked  between  the  experimental  and  control  groups  to  see  the 
influence  of  the  interventions.  It  would  be  interesting  to  analyze  the  trust  development  within  the 
teams  and  roles.  This  could  give  a  measure  of  direction  of  trust  development.  For  instance,  is 
trust  development  related  to  the  role  of  the  team  members?  It  may  be  that  those  in  leadership 
tend  to  trust  more  or  less  than  those  that  they  lead.  This  could  involve  more  roles  and 
relationships  than  commander  and  ground  solider  and  thus  larger  teams.  This  type  of  study 
would  produce  more  interactions  upon  which  to  rate  trust  in  different  team  members.  Also,  the 
differences  between  populations  (specifically  military  versus  civilian)  should  be  explored.  There 
may  be  aspects  of  training  that  may  influence  the  levels  and  development  rates  of  trust  that 
military  participants  possess. 

One  of  the  shortcomings  of  this  research  was  that  the  scenario  may  have  been  too  short. 
The  duration  of  the  task  could  be  manipulated  to  ensure  that  team  members  interact  with  each 
other  and  have  time  to  assess  teammates  and  develop  clearer  impressions  of  the  team.  This  may 
be  key  to  observing  the  desired  variability  in  distrust/trust  levels.  This  could  be  done  with 
multiple  trials  of  the  same  teams. 

There  were  no  significant  differences  between  the  groups  for  these  interventions.  Several 
researchers  have  proposed  a  number  of  other  mitigation  strategies  for  virtual  teams.  Some  of 
these  could  be  identified  as  candidate  trust  facilitating  interventions  for  virtual  teams  in  future 
studies  as  well. 

This  study  has  revealed  a  very  worthwhile  area  of  research.  Many  SRL  tasks  are 
completed  virtually  by  remote  teams.  This  research  area  can  provide  baseline  requirements  for 
tools  and  procedures  to  allow  for  effective  completion  of  these  types  of  tasks. 
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3.8 


Appendixes 

Appendix  A — Gameplay  Instructions 


3.8.1 


Welcome  to  the  Virtual  Teams  Study.  All  participants  will  receive  $10  for  taking  part  in  this  study.  Teams  that 
complete  all  the  missions  effectively  and  efficiently  will  receive  an  additional  $10  at  the  completion  of  the  study. 

Please  read  these  and  other  instructions  carefully  to  increase  your  chances  of  obtaining  the  additional  $10. 

To  your  right  you  will  find  a  key  map,  which  list  the  functions  each  key  is  used  for  within  the  VBS  1  simulation. 

The  following  will  walk  you  through  a  brief  tutorial,  which  will  allow  you  to  better  familiarize  yourself  with  the 
game.  This  is  a  brief  tutorial  and  should  take  no  longer  than  10  minutes  to  complete.  Please  read  though  and  familiar 
yourself  with  the  game  in  a  timely  fashion. 

Team  Communication: 

To  communicate  with  your  team  press  /  .  Once  you  have  typed  your  message,  simply  press  ENTER  to  send  your 
message  to  your  teammates. 

There  are  also  several  chat  channels,  which  can  be  selected  by  using  the  comma  ( , )  and  period  ( .  )  keys.  During 
this  study  it  is  not  necessary  to  change  the  channel  you  are  speaking  in,  but  if  you  accidentally  change  the  channel 
you  can  use  the  comma  and  period  to  get  back  to  the  default  channel,  which  is  the  SIDE  CHANNEL. 

If  at  any  time  you  have  a  question  during  this  tutorial,  simply  send  a  message  saying  “Help”  and  someone  will  come 
to  answer  your  question. 


Type,  “I  have  begun  the  training”  now. 


Looki 

ng 


around  the  environment: 

To  begin,  move  the  mouse  cursor  around.  The  mouse  is  used  to  control  your  view,  the  direction  you  are  moving  in, 
and  to  steer  vehicles.  To  look  up,  move  the  mouse  forward,  to  look  down,  move  the  mouse  backward.  To  turn  your 
view  left  or  right,  or  to  steer  a  vehicle  to  the  left  or  right,  simply  move  the  mouse  in  that  direction. 


Try  looking  around  the  environment  now. 


Basic  Movement: 

Movement  is  done  with  the  W,  A,  S,  D  keys,  similar  to  most  First-Person  Shooters.  W  will 
move  your  character  forward  or  accelerate  your  vehicle;  S  will  move  your  character  backward, 
stop  your  vehicle,  or  if  held,  move  your  vehicle  in  reverse.  A  will  make  your  character  side  step 
to  the  left,  or  turn  your  vehicle  to  the  left.  D  will  make  your  character  side  step  to  the  right,  or 
turn  your  vehicle  to  the  right. 

When  driving  a  vehicle,  it  is  recommended  that  you  use  the  mouse  to  steer,  as  the  A  and  D  keys 
can  sometimes  be  unresponsive.  To  run,  simply  press  E,  or  hold  the  SHIFT  key  while  pressing 
W.  This  method  also  works  when  driving  a  vehicle,  and  will  make  you  accelerate  beyond  the 
normal  driving  speed. 


t 


W 

<- 
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There  are  also  several  standing  positions  your  character  can  take,  such  as  crouching  or  going  prone  (laying  down). 
To  crouch,  simply  press  Q,  to  stand  up  again,  press  Q  once  more.  To  go  prone,  press  Z,  and  press  Z  again  stand  up. 


Try  out  the  movement  controls  now. 


Vehic 
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le  Use: 

To  enter  a  vehicle,  walk  up  to  the  driver  side  door  and  a  menu  will  appear  in  the  bottom  right  corner.  In  this  menu 
you  are  presented  with  several  options  including: 

•  get  in  the  vehicle  as  the  driver  (get  in  as  driver), 

•  ride  as  a  passenger  (ride  in  back), 

•  enter  the  gunner  positions  (get  in  as  gunner). 

Using  the  bracket  keys  (  [  and  ]  ),  you  can  move  the  selector  through  the  list  of  choices  available  in  the  list.  Once 
you  have  highlighted  the  position  you  with  to  enter  in  a  given  vehicle,  press  ENTER  on  the  keyboard  to  enter  that 
positions.  If  for  any  reason  this  menu  does  not  appear,  simply  press  one  of  the  bracket  keys  and  it  should  appear. 
When  inside  a  vehicle,  coming  to  a  stop  or  pressing  one  of  the  bracket  keys  will  bring  up  the  commands  menu  in  the 
bottom  right  corner.  To  exit  a  vehicle,  highlight  the  “get  out'’  option,  and  press  ENTER  on  the  keyboard. 

Some  vehicles  contain  weapons  and  items  such  as  grenades  and  binoculars.  These  items  are  considered  off  limits  to 
your  squad,  and  removing  them  from  vehicles  will  result  in  a  mission  failure. 

Once  you  are  inside  a  vehicle,  you  can  press  the  ENTER  key  on  the  Number  Pad  to  switch  to  a  third-person  view. 
This  view  makes  driving  certain  vehicles  much  easier,  and  can  be  used  at  your  own  discretion. 


This 
complet 
es  the 
VBS  1 

familiarization.  When  you  have  driven  to  the  airport  and  exited  your  vehicle  please  wait  for  further  instructions. 


Enter  one  of  the  humvees  in  your  area  and  try  out  these  controls  now.  When  you  are  comfortable 
with  the  controls  and  driving  within  the  game,  drive  to  the  airport  and  exit  the  vehicle. 
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3.8.2  Appendix  B — Key  Map 


Team  Communication 

Key _ 

/ 

enter 
.  (period) 


Basic  Movement 

Key _ 

W 

s 

A 

D 

E 

Q 

z 

ENTER  (on  numpad) 
M 


Action 

Opens  the  chat  dialogue  where  you  can  type  what  you  want  to  say 
Sends  the  communication  you  just  typed  to  all  in  the  channel  used 
Changes  chat  channel 

t 


Action 

Move  forward 
Move  backward 
Step  left 
Step  right 

Run  forward  * 

Move  to  a  crouched  position 
Press  once  to  go  prone  (lay  down)  /  press  again  to 
stand  up 

Changes  view  perspective  from  1st  person  to  3ld  person. 
Brings  up  the  map 


W 

A 

s 

D 

Vehicle  Use 

Key _ Action 


W 

E 

S 

A 

D 

Mouse 


Accelerate 
Fast  accelerate 
Brake  /  continue 
Turn  left 
Turn  right 
Look  where  you 


holding  to  move  in  reverse 


wish  to  drive  and  the  car  will  turn 


Command  Menu 

Key _ Action 


|  Cycle  upwards  through  items  in  command  menu 

|  Cycle  downwards  through  items  in  command  menu 

Return  Select  highlighted  item  in  command  menu 
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3.8.3  Appendix  C — Situation  Awareness  Calibration 


It  is  key  that  the  teams  maintain  a  shared  awareness  of  the  current  situation  throughout  the  scenario.  Each  player 
will  have  information  that  the  others  do  not.  In  order  to  work  effectively  as  a  team,  it  is  important  that  each  play 
convey  key  information  to  the  others,  while  avoiding  overloading  communication  channels  with  extraneous  chat. 

In  this  scenario,  you  will  be  asked  to  use  a  situation  awareness  calibration  strategy  to  facilitate  information  sharing 
without  overloading  communication  channels.  Approximately  every  10  minutes  the  commander  will  state  current 
mission  goals  for  each  soldier,  and  ask  each  soldier  to  report  on  current  status  on  assigned  mission,  enemy  activity, 
and  any  other  elements  that  put  the  mission  at  risk.  In  addition,  team  members  are  encouraged  to  share  information 
about  the  situation  and  the  mission  outside  of  the  scheduled  situation  awareness  calibration  if  they  believe  others  on 
the  team  need  the  information  immediately. 

The  commander  should  initiate  a  situation  awareness  calibration  approximately  every  10  minutes.  Soldiers  should 
expect  to  respond  to  the  commander’s  questions  every  10  minutes. 


Plea 

se 

note 
,  in 
addi 
tion 
to 
the 
situa 
tion 

awareness  calibration,  any  team  member  may  share  information  at  any  time  if  they  believe  other  members  need  the 
information  to  complete  the  mission. 


Situation  Awareness  Calibration  procedure 

1 .  Commander  states  current  mission  goals 

2.  Commander  asks  the  following  questions  of  each  soldier: 

2.1  What  is  your  current  status? 

2.2  Is  your  mission  clear? 

2.3  Have  you  seen  any  indication  of  enemy  activity? 

3.  Each  soldier  provides  commander  information  about  anything  encountered  that  might  affect  the 
mission. 
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3.8.4  Appendix  D — Commander  Information  Sheet 


Background 

The  neighboring  island  of  Al-Almar  was  recently  taken  over  by  the  head  of  the  military,  Miyindi 
Amin.  Coalition  forces  from  several  countries  are  moving  equipment  and  supplies  to  the  neighboring 
island  of  Andaman.  Y ou  are  among  the  forces  that  have  already  arrived  on  Andaman  that  are  helping 
prepare  the  military  buildup  that  will  be  required  to  take  back  Al-Almar.  The  Americans  have  amassed 
several  regiments  on  the  island  already,  and  are  now  inventorying  equipment  and  units  to  ensure  that  all 
of  the  necessary  items  are  in  position. 

Your  Role 

In  this  scenario  you  will  be  playing  the  role  of  the  Commander  of  a  logistics  squad  consisting  of 
two  soldiers,  Alpha  and  Bravo.  As  the  commander,  it  is  your  responsibility  to  assign  missions  to  both 
soldiers,  keep  records  of  units  available,  and  ensure  that  enough  units  are  present  on  the  island  for  the 
upcoming  conflict.  You  will  receive  Mission  Orders  from  headquarters  sporadically,  and  it  is  your 
responsibility  to  ensure  that  your  soldiers  complete  the  tasks  accurately  and  in  a  timely  fashion. 

It  is  important  for  you  to  remain  in  your  logistics  role,  as  you  and  your  team  are  not  properly 
equipped  to  be  members  of  a  combat  unit.  It  is  critical  that  coalition  forces  avoid  escalating  hostilities  in 
the  area.  Any  hostile  actions  initiated  by  you  or  your  logistics  team  members  will  be  considered  a  mission 
failure. 

Your  Team 

The  soldiers  you  are  working  with  have  recently  deployed  and  are  extremely  inexperienced. 

They  may  have  difficulty  following  the  terrain  using  the  provided  map,  and  recognizing  the  equipment 
they  are  asked  to  report  on.  It  is  critical  that  you,  as  commander,  provide  accurate  information  up  the 
chain  regarding  resources  available.  Y ou  must  decide  whether  the  intel  provided  with  the  mission  or  the 
eyes-on-report  from  the  alpha  and  bravo  soldiers  is  most  reliable. 

Reports 

Attached  to  each  Mission  Order  is  a  Mission  Report.  These  reports  will  be  sent  back  to 
headquarters,  along  with  your  final  count  of  units,  to  determine  what  resources  are  needed.  Mission 
Reports  also  have  a  space  for  additional  information.  In  this  area  you  are  free  to  write  down  any  mission 
critical  information  that  may  be  reported  to  you  during  the  course  of  a  mission,  such  as  enemy  contact. 

See  the  attached  sheet  for  an  example  of  how  to  fill  out  a  Mission  Report. 

Mission  Checklist 

Along  with  the  example  Mission  Report,  you  will  also  find  attached  to  the  back  of  this  document 
your  Resource  Checklist.  This  document  is  to  be  used  by  you  throughout  the  scenario  to  keep  track  of 
equipment  and  supplies.  In  the  “Accounted  For”  column  you  will  indicate  the  number  of  each  unit  type 
accounted  for.  In  the  “Number  Requested”  column  you  will  indicate  the  number  of  additional  units 
requested  (if  any)  to  meet  the  mission  needs.  After  the  scenario  is  complete,  you  will  turn  in  the  Resource 
Checklist,  with  your  final  counts  for  equipment  and  the  number  of  units  you  still  need  to  meet  the  mission 
goals. 

Begin  Play 

After  you  have  reviewed  the  example  Mission  Report  and  feel  comfortable  with  your  position  as 
commander,  send  a  message  to  your  team  stating  “I  am  your  new  commander,  I  will  be  issuing  you  orders 
from  now  on.”  Once  you  have  done  this  Mission  Orders  will  begin  to  arrive. 
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3.8.5  Appendix  E — Resource  Checklist 


Unit  Type 

Accounted  For 

Total  Needed 

Number  Requested 

Fuel  Truck 

2 

Ammo  Truck 

3 

Troop  Truck 

1 

Bradley  Fighting  Vehicle 

4 

Tank 

6 

Helicopter 

4 

Soldier 

45 

Anti-Tank  Soldier 

6 

Medic 

6 

Officer 

1 

Humvee 

3 

Ambulance 

2 

Use  this  form  to  track  resources  throughout  the  scenario.  This  page  will  be  turned  in  at  the  end  of  the 
scenario  with  your  finally  tally  count. 

3. 8. 6  Appendix  F — Example  Mission  Order 

Order  0 

SEND  BRAVO  SQUAD  TO  DJEBEL  GABR 

MISSION:  COUNT  NUMBER  OF  HUMVEES  PRESENT  AT  DJEBEL  GABR.  ACCURATE  COUNT  IS 
CRUCIAL  TO  MISSION  SUCCESS. 

INTEL:  RECENT  SATELLITE  IMAGERY  SHOWS  SIX  HUMVEES  CURRENTLY  PRESENT  IN  THE  TOWN. 

3.8.7  Appendix  G — Alpha  Soldier  Information  Sheet 


The  neighboring  island  of  Al-Almar  was  recently  taken  over  by  the  head  of  the  military,  Miyindi  Amin. 

Coalition  forces  from  several  countries  are  moving  equipment  and  supplies  to  the  neighboring  island  of 
Andaman.  You  are  among  the  forces  that  have  already  arrived  on  Andaman  that  are  helping  prepare  the  military 
buildup  that  will  be  required  to  take  back  Al-Almar.  The  Americans  have  amassed  several  regiments  on  the  island 
already,  and  are  now  inventorying  equipment  and  units  to  ensure  that  all  of  the  necessary  items  are  in  position. 

In  this  scenario  you  will  be  playing  the  role  of  soldier  Alpha,  who  is  part  of  a  team  of  logistics  support 
staff.  You  and  your  teammates  are  responsible  for  locating  and  verifying  intelligence  reports  on  the  number  of  units 
or  pieces  of  equipment  that  are  on  the  island.  When  you  receive  a  mission  from  your  commander,  drive  to  the  stated 
location,  perform  the  mission  task,  report  back  your  findings  to  the  commander,  and  then  wait  for  further  orders. 

It  is  important  to  remember  that  you  are  not  equipped  to  engage  enemy  contacts.  If  for  any  reason  you 
come  in  contact  with  an  enemy,  it  is  best  to  leave  the  area  as  quickly  as  possible.  It  is  critical  that  coalition  forces 
avoid  escalating  hostilities  in  the  area.  Any  hostile  actions  initiated  by  you  or  any  other  members  of  your  logistics 
team  will  be  considered  a  mission  failure. 
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3. 8. 8  Appendix  H — Us  Army  Reference  Sheet 


Troop  Transport 


Dignitary 

m 
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3. 8. 9  Appendix  I — Enemy  Reference  Sheet 


Enemy  Forces  Reference  Sheet 
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3.8.10  Appendix  J— Mission  Orders  And  Delivery 


Order  1 


SEND  ALPHA  SQUAD  TO  DAR  AL-HARB 

MISSION:  COUNT  NUMBER  OF  FUEL  TRUCKS  PRESENT  IN  DAR  AL-HARB. 
ACCURATE  COUNT  IS  CRUCIAL  TO  MISSION  SUCCESS. 

INTEL:  RECENT  SATELLITE  IMAGERY  SHOWS  TWO  FUEL  TRUCKS  CURRENTLY 
PRESENT  IN  THE  TOWN. 


Order  1 


SEND  BRAVO  SQUAD  TO  HARG 

MISSION:  COUNT  NUMBER  OF  AMBULANCES  PRESENT  IN  HARG. 
ACCURATE  COUNT  IS  CRUCIAL  TO  MISSION  SUCCESS. 

INTEL:  RECENT  SATELLITE  IMAGERY  SHOWS  TWO  AMBULANCES 
CURRENTLY  PRESENT  IN  THE  TOWN. 


Order  2 


SEND  ALPHA  SQUAD  TO  HARG 

MISSION:  COUNT  NUMBER  OF  AMMO  TRUCKS  PRESENT  IN  HARG.  ONE 
AMMO  TRUCK  MUST  BE  RETURNED  TO  THE  AIRPORT.  ACCURATE  COUNT 
IS  CRUCIAL  TO  MISSION  SUCCESS. 

INTEL:  RECENT  SATELLITE  IMAGERY  SHOWS  THREE  AMMO  TRUCKS 
CURRENTLY  PRESENT  IN  THE  TOWN. 


Order  2 


SEND  BRAVO  SQUAD  TO  THE  NORTH  OF  DAR  AS-SUTH 

MISSION:  COUNT  NUMBER  OF  TANKS  PRESENT  IN  DAR  AS-SUTH. 
ACCURATE  COUNT  IS  CRUCIAL  TO  MISSION  SUCCESS. 

INTEL:  RECENT  SATELLITE  IMAGERY  SHOWS  TWO  TANKS  CURRENTLY 
PRESENT  IN  THE  TOWN. 
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Order  3 


SEND  ALPHA  SQUAD  TO  THE  SADH  OUTPOST. 

MISSION:  COUNT  NUMBER  OF  SOLDIERS  PRESENT  AT  THE  SADH  OUTPOST. 
WE  MUST  ALSO  KNOW  THE  NUMBER  OF  ANTI-TANK  SOLDIERS  PRESENT 
AMONG  THE  SOLDIERS  AT  THE  OUTPOST.  ACCURATE  COUNT  IS  CRUCIAL 
TO  MISSION  SUCCESS. 

ONCE  ALL  SOLDIERS  HAVE  BEEN  ACCOUNTED  FOR,  A  FUEL  TRUCK 
PRESENT  AT  THE  OUTPOST  NEEDS  TO  BE  TRANSPORTED  TO  DJEBEL  GABR. 

INTEL:  RECENT  SATELLITE  IMAGERY  SHOWS  FIFTEEN  SOLDIERS,  WITH 
THREE  ANTI-TANK  SOLDIERS  AMONG  THEM  CURRENTLY  PRESENT  IN  THE 
TOWN. 


Order  3 

SEND  BRAVO  SQUAD  TO  MUT 

MISSION:  COUNT  NUMBER  OF  SOLDIERS  PRESENT  IN  MUT.  ACCURATE 
COUNT  IS  CRUCIAL  TO  MISSION  SUCCESS. 

INTEL:  RECENT  SATELLITE  IMAGERY  SHOWS  TWENTY-SIX  SOLDIERS 
CURRENTLY  PRESENT  IN  THE  TOWN. 


Order  4 


SEND  ALPHA  SQUAD  TO  DJEBEL  GABR. 

MISSION:  COUNT  NUMBER  OF  HELICOPTERS  AND  HUMVEES  PRESENT  IN 
DJEBEL  GABR.  ACCURATE  COUNT  IS  CRUCIAL  TO  MISSION  SUCCESS. 

INTEL:  RECENT  SATELLITE  IMAGERY  SHOWS  THREE  HELICOPTERS  AND 
THREE  HUMVEES  CURRENTLY  PRESENT  IN  THE  TOWN. 


Order  4 


SEND  BRAVO  SQUAD  TO  THE  SADH  OUTPOST 

MISSION:  COUNT  NUMBER  OF  OFFICERS  PRESENT  AT  THE  SADH  OUTPOST. 
ACCURATE  COUNT  IS  CRUCIAL  TO  MISSION  SUCCESS. 

INTEL:  RECENT  SATELLITE  IMAGERY  SHOWS  ONE  OFFICER  CURRENTLY 
PRESENT  IN  THE  TOWN. 


78 


Order  5 


SEND  ALPHA  SQUAD  TO  DJEBEL  GABR. 

MISSION:  TRANSPORT  A  SOLDIER  TRUCK  FROM  DJEBEL  GABR  TO  MUT. 
ACCURATE  COUNT  IS  CRUCIAL  TO  MISSION  SUCCESS. 

INTEL:  RECENT  SATELLITE  IMAGERY  SHOWS  ONE  TRANSPORT  TRUCK 
CURRENTLY  PRESENT  IN  THE  TOWN. 


Order  5 

SEND  BRAVO  SQUAD  TO  DAR  AL-HARB 

MISSION:  COUNT  NUMBER  OF  BRADLEY  FIGHTING  VEHICLES  PRESENT  AT 
DAR  AL-HARB.  ACCURATE  COUNT  IS  CRUCIAL  TO  MISSION  SUCCESS. 

INTEL:  RECENT  SATELLITE  IMAGERY  SHOWS  FOUR  BRADLEY  FIGHTING 
VEHICLES  CURRENTLY  PRESENT  IN  THE  TOWN. 


Order  6 

SEND  ALPHA  SQUAD  TO  MUT. 

MISSION:  COUNT  THE  NUMBER  OF  MEDICS  PRESENT  IN  MUT.  ACCURATE 
COUNT  IS  CRUCIAL  TO  MISSION  SUCCESS. 

INTEL:  RECENT  SATELLITE  IMAGERY  SHOWS  FIVE  MEDICS  CURRENTLY 
PRESENT  IN  THE  TOWN. 


Order  6 

SEND  BRAVO  SQUAD  TO  DJEBEL  GABR 

MISSION:  COUNT  NUMBER  OF  TANKS  PRESENT  AT  DJEBEL  GABR. 
ACCURATE  COUNT  IS  CRUCIAL  TO  MISSION  SUCCESS. 

INTEL:  RECENT  SATELLITE  IMAGERY  SHOWS  FOUR  TANKS  CURRENTLY 
PRESENT  IN  THE  TOWN. 
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TOP  PRIORITY  MISSION  REQUEST 


A  Cl 30  HAS  JUST  CRASHED  OUTSIDE  OF  MUT.  SEND  ALPHA  SQUAD  TO 
CHECK  FOR  SURVIVORS. 

MISSION:  SEARCH  FOR  SURVIVORS  AT  THE  CRASH  SITE.  ACCURATE 
COUNT  IS  CRUCIAL  TO  MISSION  SUCCESS. 

INTEL:  THERE  IS  NO  INTEL  AT  THIS  TIME. 


Order  7 

SEND  BRAVO  SQUAD  TO  THE  NORTH  OF  DAR  AS-SUTH 

MISSION:  FIND  AND  PROTECT  THE  DIGNITIARIES  OUTSIDE  OF  DAR  AS- 
SUTH. 

INTEL:  RECENT  SATELLITE  IMAGERY  SHOWS  TWO  DIGNITIARIES 
CURRENTLY  PRESENT  IN  THE  TOWN. 


Order  7 

SEND  ALPHA  SQUAD  TO  THE  NORTH  OF  DAR  AS-SUTH. 

MISSION:  LOCATE,  COUNT,  AND  GUARD  THE  DIGNITIAIRES  PRESENT 
OUTSIDE  OF  DAR  AS-SUTH.  ALPHA  SQUAD  MUST  MEET  UP  WITH  BRAVO 
SQUAD  AND  GUARD  THE  DIGNITIES.  ACCURATE  COUNT  IS  CRUCIAL  TO 
MISSION  SUCCESS. 

INTEL:  RECENT  SATELLITE  IMAGERY  SHOWS  TWO  DIGNITIARIES 
CURRENTLY  PRESENT  IN  THE  TOWN. 
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Delivery  times 

Time  Order  Number  for  Alpha  and  Bravo 


4:00 

5:30 

10:00 

11:10 

14:20 

16:20 

16:30 

18:20 

19:40 

22:00 

23:20 

25:20 

27:40 


1  Alpha,  1  Bravo 

2  Alpha 

2  Bravo 

3  Alpha 

3  Bravo 

4  Bravo 

5  Bravo 

4  Alpha 

5  Alpha 

6  Bravo 

6  Alpha 

IMMEDIATE  MISSION 

7  Bravo 
7  Alpha 
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4.1  Executive  Summary 


The  purpose  of  this  task  is  to  develop  new  forecasting  techniques  that  can  be  readily  applied 
within  the  military  context  (especially  sparing  forecasts),  evaluate  the  effectiveness  of  those 
techniques,  and  to  examine  their  operational  effects  within  a  supply  chain  setting.  A  new 
technique  call  MCARTA  was  developed  and  shown  to  be  better  than  other  known  techniques  for 
intermittent  demand  based  on  extensive  empirical  testing.  A  number  of  competing  metrics  were 
examined  for  comparing  forecasting  techniques.  In  addition,  a  simulation  environment  was 
developed  to  allow  the  evaluation  of  the  effects  of  using  different  forecasting  techniques  within 
multi-echelon  spare  parts  networks. 

The  forecasting  of  logistics  demand  (inventory  requirements,  transportation  loads,  etc.)  has 
been  a  perennial  problem  for  military  logisticians.  While  many  of  the  forecasting  methods  used 
in  commercial  practice  can  be  utilized  in  a  military  setting,  military  logistical  requirements  have 
characteristics  that  necessitate  the  development  of  specialized  or  unique  forecasting  techniques. 
While  cost  is  always  a  concern,  the  operational  metrics  in  military  supply  chains  are  very 
different  than  in  a  commercial  setting.  In  a  military  planning  situation,  it  is  the  ability  to 
complete  the  mission,  which  is  paramount.  Standard  methodologies  for  evaluating  forecasting 
techniques  involve  fitting  models  to  historical  data  and  selecting  the  parameters  of  the  models 
and  the  “optimal”  technique  based  on  minimizing  forecasting  error.  Previous  research  by  the  PI 
and  co-PI  for  the  Navy  (“ Evaluation  of  Intermittent  Forecasting  Techniques ”)  and  Air  Force 
(“C/KC-135  Weapon  System  Stockage  Policy  and  MICAP  Avoidance  Analysis  ”)  has  indicated  a 
number  of  problems  that  need  to  be  addressed: 
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•  Intermittent  demand  that  is  generated  within  spare  part  modeling  contexts  is  extremely 
difficult  to  forecast  and  existing  techniques  remain  inadequate. 

•  Should  different  forecasting  models  be  used  for  each  and  every  item  or  can  a  standard  set  of 
techniques  be  identified  that  are  operationally  robust  across  the  supply  chain?  For  example, 
when  forecasting  part  demand,  it  becomes  computationally  problematic  to  fit  an  optimal 
forecasting  model  to  hundreds  of  thousands  of  items.  Can  preprocessing  techniques  be 
developed  that  are  strong  predictors  of  the  most  operational  robust  technique? 

•  No  standardized  methodology  exists  for  evaluating  the  effectiveness  of  forecasting 
techniques  within  an  operational  context.  Such  a  methodology  would  assist  military  planners 
in  evaluating  whether  or  not  it  is  worth  improving  the  forecasting  in  a  certain  area  and  then  in 
selecting  the  most  operationally  robust  forecasting  technique. 

The  task  was  divided  into  four  sub-topics:  1)  new  intennittent  demand  forecasting,  2)  new 
metrics  for  evaluating  forecasts,  3)  new  methods  of  applying  techniques  to  large  scale  inventory 
systems,  and  4)  operational  effects  of  inventory  systems  with  forecast  based  policy  updating. 
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4.2  The  MCARTA  Technique  for  Intermittent  Demand  Forecasting 

4.2.1  Problem/Background 

Intermittent  demand  is  characterized  by  demand  data  that  has  many  time  periods  with 
zero  demands.  It  is  hard  to  model  intermittent  demand  by  conventional  distributions.  Intennittent 
demand  is  commonly  found  in  military  supply  network  among  reparable  spare  parts.  The 
availability  of  spare  parts  that  depends  on  the  accuracy  of  forecast,  affects  the  mission 
accomplishment  in  military.  The  purpose  of  this  research  is  to  develop  an  intennittent  demand 
forecasting  technique  named  MCARTA  based  on  a  Markov  Chain  model  and  ARTA  algorithm. 
The  first  part  of  this  research  investigates  the  best  parameter  settings  for  the  MCARTA 
technique.  In  the  second  part  of  the  research,  various  intennittent  forecast  techniques  are 
compared  with  MCARTA  and  the  robustness  of  each  of  the  forecasting  techniques  under  the 
various  demand  scenarios  were  investigated.  The  other  forecasting  techniques  considered  for  the 
research  are  simple  exponential  smoothing,  moving  average,  cumulative  average,  Croston, 
Syntetos,  and  naive  forecast  1  (NF1). 

4.2.2  Methods,  Assumptions,  and  Procedures 

MCARTA  is  a  bootstrapping  approach  for  making  a  forecast  estimate.  It  requires 
estimates  for  the  parameters  (called  the  statistical  parameters  of  MCARTA) :  probability  of  non¬ 
zero  demand  after  zero  demand,  probability  of  non-zero  demand  after  non-zero  demand,  mean  of 
non-zero  demand,  variance  of  non-zero  demand  and  lag  1  correlation  coefficient  of  non-zero 
demand. 

Based  on  the  transition  probabilities  a  binary  generator  is  created,  in  which  the  zero 
represents  zero  forecast  estimates  and  one  represents  non-zero  forecast  estimates.  At  each  time 


85 


index,  the  underlying  model  is  a  Markov  chain.  With  the  decision  made  by  the  binary  generators, 
MCARTA  will  generate  non-zero  forecast  estimates.  The  nonzero  forecast  is  generated  based  on 
the  ARTA  (Autoregressive  to  Anything)  algorithm  with  an  underlying  non-zero  positive 
distribution  (e.g.  Poisson,  Negative  Binomial,  Binomial,  etc.)  selected  based  on  the  mean  and  the 
variance  of  the  non-zero  demand. 

For  the  experiments,  demand  series  that  are  representative  of  intermittent  demand  were 
generated  and  forecasts  made  at  different  parameter  levels  of  MCARTA.  At  the  end  of  the  run, 
the  different  forecast  errors  (MAD,  MSE,  mean  unbiased  absolute  percentage  error,  relative 
mean  absolute  error,  and  mean  absolute  scaled  error)  are  estimated;  bias  and  variance  are  also 
computed.  The  demand  scenario  is  replicated  and  the  forecast  errors,  bias  and  variance  at  each 
replication  measured.  The  best  parameter  settings  of  MCARTA  were  selected  using  a  multiple 
comparison  with  the  best  approach  based  on  forecast  errors.  The  bias  and  the  variance  were 
statistically  analyzed  and  inferences  made  on  the  statistical  quality  of  the  forecast  estimates  of 
the  best  MCARTA.  The  research  also  discusses  an  intermittent  demand  generator  which 
generated  demand  according  to  user  specified  levels  of  demand  characteristics.  The  demand 
generator  is  based  on  Batch  -  on/off  algorithm.  We  considered  40  different  levels  of  demand 
scenarios. 

4.3  Results  and  Discussion 

The  best  MCARTA  settings  were  selected  based  on  forecast  errors.  A  bootstrap  size  of 
1000  is  recommended  along  with  the  use  of  the  mean  of  bootstrap  sample  as  the  bootstrap 
estimator.  The  whole  demand  series  should  be  used  to  estimate  the  statistical  parameters  of 
MCARTA.  The  best  settings  of  updating  the  statistical  parameters  selected  were:  update  the 
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statistical  parameter  continuously  and  the  ARTA  matching  may  be  turned  off.  The  bias  and  the 
variance  of  the  best  MCARTA  were  statistically  analyzed.  The  MCARTA  appears  to  have  low 
bias,  and  was  shown  to  have  statistically  similar  bias  properties  as  the  next  best  technique  due  to 
Syntetos().  We  also  see  that  the  bias  of  MCARTA  and  the  Syntetos  are  statistically  equal.  The 
variance  of  MCARTA  is  lower  compared  to  that  of  the  Syntetos  technique.  These  properties  of 
MCARTA  qualify  it  not  only  as  a  very  competent  intennittent  demand  forecasting  technique  but 
also  as  one  of  the  best.  In  addition,  it  is  seen  that  the  new  demand  generator  is  efficient  and  can 
be  fully  controlled  by  the  user,  which  is  another  contribution  of  the  research. 

Using  multiple  comparisons  with  the  best,  based  on  MAD,  relative  mean  absolute  error, 
and  mean  absolute  scaled  error,  MCARTA  is  clearly  the  best  intermittent  forecasting  technique. 
Also,  based  on  MSE  and  mean  unbiased  absolute  percentage  error,  MCARTA  is  as  competent  as 
the  best  technique.  It  can  be  inferred  from  this  result  that  MCARTA  is  robust  across  different 
demand  scenarios.  The  performance  the  forecasting  techniques  across  specific  demand  scenarios 
was  also  investigated.  We  observed  the  demand  scenarios  that  represent  the  erratic  demand, 
intermittent  demand,  slow  demand,  bursty  demand,  and  that  of  an  US  Navy  inventory  system 
that  holds  reparable  spare  parts  with  intermittent  demand.  We  observed  that  in  most  of  the  cases 
MCARTA  was  the  best  technique  and  other  cases  MCARTA  is  as  competent  as  the  best 
technique.  All  these  comparisons  are  based  on  each  of  the  forecast  error  metrics. 
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4.4  The  Effects  of  Forecasting  Techniques  on  the  Operational  Performance  Measures  of  an 
Inventory  System 

4.4.1  Problem/Background 

Traditionally,  in  a  real  inventory  system  the  policy  may  be  updated  based  on  the  forecast 
estimates  and  the  forecast  error  from  a  forecasting  technique  selected  by  inventory  managers. 
There  are  several  research  studies  that  justify  that  the  selection  of  the  forecasting  technique 
affects  the  operational  performance  measures  (OPM)  of  the  inventory  system.  This  motivated  the 
investigation  of  whether  there  is  a  relationship  between  forecast  error  (FE)  and  OPM. 

Using  the  forecast  estimate  and  forecast  error  inventory  managers  model  the  lead  time 
demand  distribution  (FTDD).  Inventory  optimization  models  that  depend  on  the  approximate 
FTDD  are  used  to  set  policy  parameters.  Approximations  occur  during  this  process  because  an 
assumed  family  of  distributions  (e.g.  Neg-binomial)  is  used  for  the  lead  time  demand  and  also 
owing  to  the  estimation  of  the  mean  and  variance  of  the  lead  time  demand  distribution.  How  well 
the  approximate  lead  time  demand  distribution  fits  the  actual  lead  time  demand  distribution  is 
critical  to  how  well  the  planned  for  parameter  settings  allow  for  the  inventory  model 
performance  to  match  the  actual  operational  performance.  Thus,  rather  than  use  forecast  error  to 
select  the  forecasting  technique,  a  metric  based  on  the  goodness  of  the  approximate  lead  time 
demand  distribution  should  be  used.  This  research  investigates  a  new  performance  metric  for 
picking  the  best  forecasting  technique. 

4.4.2  Methods,  Assumptions,  and  Procedures 

A  (r,  Q)  inventory  system  which  encounters  intermittent  demand  was  simulated  for  the 
research.  The  demand  arrives  and  it  is  filled.  The  system  requests  replenishment  and  the 


replenishments  are  fulfilled  with  a  delay.  The  demand  during  the  lead  time  of  replenishment  is 
captured.  At  the  end  of  the  run,  the  empirical  LTDD  is  built.  Meanwhile,  during  the  run, 
forecasts  are  made  based  on  the  relevant  forecasting  technique:  MCARTA,  SES,  MA,  CA, 
Croston,  Syntetos,  and  NF 1 .  The  forecast  technique  makes  a  forecast  for  the  next  period.  At  the 
end  of  the  run,  we  observe  the  next  period  forecast  which  is  an  estimate  for  the  demand  per  unit 
time  (DPUT).  At  the  end  of  the  run,  we  estimate  the  variance  of  the  forecast  via  the  mean 
squared  error,  an  estimate  for  the  variance  of  the  DPUT.  This  is  matched  with  the  first  2 
moments  of  the  hypothesized  LTDD.  The  hypothesized  LTDD  is  also  built  based  on  next  period 
forecast  estimate  and  the  actual  variance  from  the  observed  LTD.  The  goodness  of  fit  of  the 
hypothesized  LTDD  from  each  forecast  technique  is  compared  with  the  empirical  LTDD.  The 
goodness  of  fit  is  measured  based  on  Kullback-Leibler  (KL)  divergence  measure,  Anderson 
Darling  test  statistics,  Kolmogorov  Smirnov  test  statistics  and  sum  of  squared  error.  We  study 
the  relationship  between  forecast  error  and  these  metrics  via  the  Kendall’s  Tau  correlation 
coefficient. 

4. 4. 3  Results  and  Discussion 

Preliminary  research  shows  that  based  on  the  LTDD  metrics,  MCARTA  is  as  competent  as  other 
forecasting  techniques. 
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4.5  A  Meta  Forecasting  Methodology  for  Large  Scale  Inventory  Systems  with  Intennittent 

Demand 

4.5.1  Problem/Background 

This  research  presents  a  meta-forecasting  approach  for  recommending  the  most 
appropriate  forecasting  technique  for  an  intennittent  demand  series  based  on  a  multinomial 
logistic  regression  classifier.  The  meta-forecaster  is  based  on  a  mapping  between  a  demand 
attribute  space  and  the  best  forecasting  technique.  The  demand  attribute  space  is  based  on  the 
estimates  from  the  demand  series  of  the  following  attributes:  probability  of  non-zero  demand 
after  zero  demand,  probability  of  non-zero  demand  after  non-zero  demand,  mean  demand, 
demand  variance,  lag  1  correlation  coefficient  of  demand,  mean  interval  between  non-zero 
demands,  variance  of  interval  between  non-zero  demands  and  lag  1  correlation  coefficient  of  the 
interval  between  non-zero  demand  and.  Based  on  the  mapping,  the  best  forecasting  technique  for 
an  unknown  demand  vector  can  be  predicted.  Given  the  demand  series,  the  demand  attributes  are 
estimated  and  then  the  classifier  is  used  to  predict  the  best  forecasting  technique. 

4.5.2  Methods,  Assumptions,  and  Procedures 

The  demand  attribute  space  for  the  meta-forecaster  is  generated  using  a  normal  to 
anything  (NORTA)  approach,  where  each  of  the  demand  attributes  are  generated  based  on  a 
marginal  distribution  that  is  representative  of  the  demand  attributes  of  intennittent  demand 
scenarios  and  at  the  same  time  are  conelated.  Thus,  a  comprehensive  representation  for  the 
demand  attribute  space  that  is  representative  of  the  intermittent  demand  was  created  for  testing. 
For  each  demand  vector,  demand  is  generated  and  forecasts  are  made  according  to  the  relevant 
forecasting  technique  and  using  a  multiple  comparison  with  the  best  technique,  the  best 
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forecasting  technique  is  selected.  The  resultant  data  are  divided  for  training  and  testing.  The 
training  data  is  trained  based  on  multinomial  logistic  regression  and  subsequently  tested. 

4.5.3  Results  and  Discussion 

In  a  pilot  study,  a  meta-forecaster  was  developed  where  the  best  forecasting  technique 
was  selected  based  on  MAD.  This  meta-forecaster  was  tested.  The  results  indicate  an  accuracy 
rate  of  70.87%  for  the  recommended  best  forecasting  technique;  and  an  87.94%accuracy  rate  for 
the  recommended  top  two  forecasting  techniques. 

4.6  Forecast  based  inventory  policy  updating 
4.6.1  Problem/Background 

The  primary  objective  of  this  research  is  to  provide  a  baseline  analysis  of  the  effect  of 
various  forecasting  techniques  on  the  effectiveness  of  METRIC/VARI-METRIC  based  sparing 
models.  This  is  achieved  via  a  simulation  approach.  This  paper  presents  an  object  oriented 
framework  that  facilitates  modeling  inventory  network  that  holds  reparable  spare  parts  whose 
METRIC /VARIMETRIC  approximation  (by  which  policies  are  updated)  is  driven  by  forecast 
estimates.  Through  the  simulation  approach  questions  regarding  the  choice  of  the  forecasting 
technique  and  the  frequency  of  updating  the  policy,  especially  in  non-stationary  demand 
scenarios  are  addressed.  The  research  also  recommends  an  appropriate  forecasting  technique  for 
this  class  of  systems.  This  research  discusses  how  the  framework  can  be  used  to  develop 
simulation  models  through  which  these  questions  can  be  addressed. 
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4.6.2  Methods,  Assumptions,  and  Procedures 

The  research  considers  the  robustness  across  several  supply-chain  configurations  of 
which  each  supply  chain  configuration  are  distinct  by  the  number  of  bases  and  the  number  of 
items.  Each  item  has  its  characteristic  budget,  demand  pattern  and  demand  rate  and  its 
corresponding  unit  cost,  reparability  at  each  location,  repair  delay  and  shipping  delay.  At  each 
supply  chain  configuration,  different  scenarios  were  considered  each  in  which  a  different 
forecasting  technique  is  assigned  to  all  the  bases  and  all  the  items.  Each  of  the  scenarios  is 
compared  with  a  base  scenario  in  which  the  corresponding  supply  chain  is  optimized  based  on 
the  best  forecasting  technique.  We  assumed  that  the  best  forecasting  technique  is  based  on  the 
demand  rate  function  that  the  user  assigned  to  that  item  in  the  simulation  model.  Hence 
comparing  each  forecasting  technique  to  best  forecasting  technique,  the  forecasting  technique 
that  is  most  close  to  the  best  forecasting  technique  across  the  different  supply  chain  configuration 
will  be  robust.  The  supply  chain  configuration  generator  is  another  contribution  of  this  research. 
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4. 6. 3  Results  and  Discussion 


In  a  pilot  study,  whose  results  are  published  in  Rossetti  et.al  (2008)  we  showed  the  ease 
in  which  the  modeling  can  be  done.  The  results  also  showed  how  the  user  can  observe  the 
improvement  in  operation  performance  measures  of  the  supply  chain  when  the  forecasting 
techniques  and  the  policy  updating  frequencies  are  changed. 
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5.1  Executive  Summary 

All  organizations  operate  in  an  atmosphere  of  uncertainty.  Despite  this  fact,  decisions 
must  be  made  that  affect  the  future  of  the  organization.  This  is  particularly  true  in  proactive 
logistics  environments  such  as  medical  logistics,  as  current  demands  at  on-going  medical 
facilities,  such  as  patient  care,  required  medicines,  and  blood/plasma  usage  must  be  carefully 
met/managed  in  the  face  of  unexpected  events,  such  as  natural  disasters,  humanitarian  aid  efforts, 
and  military  mobilizations.  Per  USTRANSCOM,  “there  is  a  lack  of  collaborative  distribution 
planning  ...  for  optimizing  the  end-to-end  distribution  process.”  In  this  project,  we  develop  a 
multi-commodity  flow  model  for  military  medical  logistics  materiel  distribution.  The  main 
characteristics  which  we  consider  include  inventory  holding  and  initial  storages,  storage  and 
commodity  flow  restrictions  and  the  supply  of  commodities  between  any  identified  facilities. 
Using  actual  data  gathered  from  medical  supplies  being  requested  in  USAMM-SWA,  we  analyze 
the  Class  VIII  supply  chain  from  prime  vendor  locations  in  the  US  all  the  way  to  a  hospital  in 
Kuwait.  Our  experimental  results  suggest  that  transportation  efficiencies  can  be  gained  via  lateral 
supply  transshipments  across  supply  chain  network  nodes  and  appropriate  inventory 
management  policies.  While  our  initial  experiments  are  not  meant  to  be  conclusive,  there  is 
evidence  that,  pending  transportation  asset  availability,  optimized  planning  and  distribution  can 
help  to  decrease  the  number  of  required  deliveries  for  class  VIII  materiel  in  theater. 
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5.2  Introduction 


All  organizations  operate  in  an  atmosphere  of  uncertainty.  Despite  this  fact,  decisions 
must  be  made  that  affect  the  future  of  the  organization.  This  is  particularly  true  in  proactive 
logistics  environments  such  as  medical  logistics,  as  current  demands  at  on-going  medical 
facilities,  such  as  patient  care,  required  medicines,  and  blood/plasma  usage  must  be  carefully 
met/managed  in  the  face  of  unexpected  events,  such  as  natural  disasters,  humanitarian  aid  efforts, 
and  military  mobilizations.  Although  new  forecasting  techniques,  software,  and  powerful 
desktop  computing  platforms  can  empower  managers  to  utilize  very  sophisticated  data  analysis 
techniques,  these  “opportunities”  for  making  effective  forecasts  are  often  missed  due  to  a  variety 
of  factors,  such  as  poor  communications  among/between  logistics  network  partners, 
improper/insufficient  training/personnel  expertise,  and  primary  reliance  on  historical  data/trends, 
to  name  only  a  few. 

As  “distribution  and  transportation  organizations  must  collaboratively  plan  both  force 
movement  and  sustainment  activities,”  (USTRANSCOM  Priority  2  document),  the  need  exists 
for  effective  planning/forecasting  capabilities  to  be  in  place  to  plan  for  future  events  while 
simultaneously  being  able  to  plan/replan  and  coordinate  the  transportation  of  materiel  and  other 
goods  for  activities  occurring  in  a  much  more  immediate  time  frame.  These  methodologies  must 
be  able  to  effectively  accommodate  “products”  that  may  have  a  finite  shelf  (useful)  life  and 
specific  due  dates  at  which  time  it  needs  to  be  delivered  to  the  appropriate  customer. 

While  many  researchers  have  conceptually  and/or  theoretical  decoupled  planning  and 
scheduling  in  academic  papers,  these  two  important  areas  often  are  very  dependent  on  one 
another.  Individual  planning  scenarios  for  achieving  one  or  more  objectives  of  interest  each 
make  their  own  (potentially  unique)  demand  on  system  resources.  Unfortunately,  current 
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operations/commitments  can  and  often  do  have  a  significant  (potentially  adverse)  effect  on  the 
feasibility  of  planning  scenarios.  Per  USTRANSCOM,  “there  is  a  lack  of  collaborative 
distribution  planning  ...  for  optimizing  the  end-to-end  distribution  process.”  Further,  “Medical 
Transportation  Managers  are  not  able  to  synchronize  load  movement  with  available  air  capacity 
when  scheduling  loads.” 

In  this  project,  we  developed  a  multi-commodity  flow  model  which  captures  many 
characteristics  of  network  design  problems.  The  main  characteristics  which  we  consider  include 
a  planning  horizon,  inventory  holding  and  initial  storages,  storage  and  commodity  flow 
restrictions  and  the  supply  of  commodities  between  any  identified  facilities  (production  plants, 
warehouses,  distribution  centers,  etc.).  We  are  able  to  bring  a  solution  for  large  size  of  problems 
due  to  the  complexity  of  the  provided  model.  In  particular,  more  useful  insights  on  these  design 
decisions  are  gained  along  with  the  progress  in  more  understating  the  nature  of  the  problem.  In 
the  current  state  of  art,  we  consider  any  flow  which  is  possible  between  facilities  and  between 
stages  (echelons).  Although  commodity  flow  is  possible  in  any  direction  between  facilities,  it  is 
only  defined  for  one  direction  between  stages.  This  network  structure  is  illustrated  in  Figure  6. 
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Figure  6.  Example  Supply  Chain  Network 


5.3  Background  Discussions  and  Motivation 

During  this  project,  research  team  members  have  met  with  numerous  Department  of 
Defense  and  support  personnel  in  the  Washington  DC  area  are  various  organizations/locations, 
such  as  the  Defense  Logistics  Agency  (DLA),  the  Logistics  Management  Institute  (LMI),  the  US 
Army’s  Telemedicine  and  Advanced  Technology  Research  Center  (TATRC),  and  prime  vendor 
Owens  &  Minor  to  learn  more  about  military  medical  logistics  and  the  Class  VIII  supply  chain. 

Lrom  these  interactions,  it  was  suggested  we  focus  our  research  efforts  on  supply  channel 
replenishment  and  distribution  from  US-based  prime  vendor  locations  to  European  and  Asian 
supply  and  distribution  centers.  In  addition,  forward-deployed  teams  and  combat  hospitals  were 
to  be  a  major  area  of  focus,  given  the  current  operations  of  the  Class  VIII  supply  chain  during 
Operation  Enduring  Freedom  and  Operation  Iraqi  Freedom  (Figure  8).  With  this  supply  chain 
network  in  mind,  we  set  forth  to  develop  optimization-based  solution  methodologies  for  planning 
the  distribution,  transshipment,  and  delivery  of  key  Class  VIII  materiel  items. 


99 


Owens/ 
MinorBalt- 
Wash  DC 


Cardinal 
East  Coast 
DC 


Germany 


Oatar 


Afghanistan 

Hosp 


Balad 

Hosp 


Kirkuk 

FDT 


Kuwait 

Hosp 


Tallil  FDT 


Tikrit 

FDT 


Iraq  / 
MLB 


Mosul 

FDT 


Figure  7.  Prime  Vendor-Driven  Supply  Chain  Network  of  Project  Focus 


5.4  Mathematical  Model  for  Class  VIII  Materiel  Distribution 

5.4.1  Sets 

N  Set  of  facilities 

V p  Set  of  facilities  at  stage  p 

K  Set  of  products 

T  Set  of  time  periods 

5.4.2  Parameters 

cl  Demand  of  product  p  at  facility  j  at  time  period  t 

c™  Transportation  cost  of  product  flow  from  facility  i  to  facility  j  by  transportation 
flow  m 

Sj,f  Initial  storage  of  product  p  at  facility  j  at  time  period  t 

hj  Unit  inventory  holding  cost  of  product  p  at  facility  j 

Ui  Upper  bound  of  inventory  amount  at  facility  i 
n  Number  of  stages 


too 


5.4.3  Variables 


f--'1  Amount  of  product  ( p  )  flow  from  facility  i  to  facility  j  at  time  period  t . 
I  jJ  Inventory  of  product  p  at  facility  j  at  time  period  t . 


5.4.4  Model  Formulation 

Total  transportation  costs  and  inventory  holding  costs  are  of  interest  in  terms  of  current 
performance  metrics.  The  following  objective  function  (1)  minimizes  the  sum  of  total 
transportation  costs  and  holding  costs. 


min 


W 


PJ  j 


(1) 


i  j  P  t  p  j  t 

As  a  main  constraint  we  consider  the  following  equality  which  satisfies  conservations  of  flows  at 
any  facilities.  We  can  consider  flow  in  commodities  as  initial  storages  and  goods  which  are 
transferred  from  some  other  facilities  in  the  same  stage  or  facilities  in  previous  stages.  On  the 
other  hand,  flow  out  commodities  can  be  thought  of  as  goods  which  are  transferred  to  some  other 
facilities  in  the  same  stage  or  facilities  in  succeeding  stages.  Finally  specified  net  amount  for  the 
corresponding  facility  constitutes  of  the  net  demand  and  the  inventory  levels.  In  this  manner,  we 
introduce  the  following  two  constraints  (2  and  3). 


2  i 

P= 1  i^VpVJVq  \{j}  T=q ksVT  uFg,  \{j} 

<2-  =  0,l,2,3,...,«  jeVq  peK 

2  I/A +  SV +  'f1 = dV + 2  Z  A-' + V 

P= 1  i^VpvjVq\{j)  T=qk&VTKjVq\{j) 

q  =  0,l,2,3,...,n  j  e  Vq  peK  te{2...l} 


(2) 


(3) 


101 


Constraint  (2)  and  (3)  are  required  that  total  flow-in  amount  is  equal  to  total  flow-out  amount. 
For  initial  period  (constraint  (2)),  there  are  no  inventory  amount  flow-in.  An  upper  bound  is 
considered  for  the  inventory,  which  is  ensured  by  the  constraint  set  (4),  while  constraint  sets  (5) 
and  (6)  are  variable  type  restriction  constraints. 


Ip-‘<ui 

l 

i  e  N,  p  e  K,t  g  T 

(4) 

o 

Al 

*£  - 

isN,  p  ^K,t  <=T 

(5) 

IV 

o 

/,  j  e  TV,  p  eK,t  gT 

(6) 

5.5  Experimental  Study 

With  the  assistance  of  US  military  personnel  at  the  Army  Medical  Department 
(AMEDD),  we  collected  the  following  list  of  the  top  20  highest  demanded  Class  VIII  products 
(lines)  at  the  United  States  Army  Medical  Materiel  Center-Southwest  Asia  (USAMMC-SWA) 
sourced  by  a  prime  vendor  (Figure  9).  In  Figure  9,  each  Class  VIII  line  is  described  and  assigned 
an  associated  ID  #  for  use  in  all  subsequent  experimentation  and  model  explanations. 

Figure  10  displays  a  hypothetical  monthly  representation  of  actual  annual  demand  levels 
by  Class  VIII  item  type  at  each  supply  chain  location  (site).  In  each  case,  AMEDD  personnel 
provided  the  research  team  with  actual  annual  demand  values  that  were  subsequently  distributed 
across  the  twelve  model  time  periods  of  interest  in  a  random  manner.  Next,  from-to  travel  times 
were  determined  by  the  research  team  using  Internet  references  and  AMEDD  personnel 
validation  (Figure  11).  This  transportation  matrix  is  expressed  in  terms  of  the  number  of  hours 
required  to  travel  between  any  two  sites.  This  information  is  important  in  the  model  as  a 

surrogate  measure  for  transportation  cost  Cy  . 
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Line  ID 

PADPREPALCOHOLIOOS  1 

B  AND  AGE_GAUZE_4 . 5  1 S  2 

C  ATHETERI  V_ANGIO_  1 S  3 

TAPE_SURGICAL_ADH_4S  4 

B  ANDAGE_ELAS_4IN_  1  OS  5 

ADHE  SI  VET  APE_  1  IN_  1 2  S  6 

GLOVELTPFLGIOOS  7 

ADHES_TAPE_SILK_2IN_6S  8 

NEEDLEHYPO 1 8GX  1.5100  9 

B  AND  AGE_EL  AS_3  IN_  1 0  S  10 

SPHYGMOM  ANOMETER_AD_  IS  11 
SYRINGEHYP05CC100S  12 

ADHESIVESKINDERM12S  13 

SYRINGEHYP0  20CC40S  14 

SYRINGEHYPOIOCCIOOS  15 

GLOVESSURG7.5 _ 100S  16 

SPONGE_4X4_TRAY_1280S  17 

BANDGE_COBAN_5X  1_30S  18 

PAD_PREP_ALC_MED_200S  19 

BANDAGE  37X37X52IN  IS  20 


Figure  8.  Top  Class  VIII  Lines  at  USAMMC-SWA  Sourced  by  Prime  Vendor 


The  initial  amount  of  inventory  for  each  Class  VIII  item  available  from  prime  vendor 
suppliers  was  calculated  by  summing  up  all  required  demand  by  item  type,  then  evenly  splitting 
this  demand  across  the  two  prime  vendor  locations  (Table  22).  Next,  in  order  to  stop  the 
optimization  model  from  simply  bulk  shipping  all  required  demanded  items  in  the  first  model 
period,  we  are  required  to  model  the  maximum  amount  of  inventory  that  could  be  stored  at  any 
site  in  any  model  time  period  (Table  23).  The  values  in  Table  23  were  computed  after  examining 
the  demand  levels  by  supply  chain  location,  although  other  methods  of  detennining  these  values 
will  also  work  in  our  model. 
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Table  22.  Annual  Demand  for  Top  20  Class  VIII  Lines— Random  Distribution  by  Month 


Month 


Line 

PAD  PREP  ALCOHOL  100S 
BANDAGE  GAUZE  4.5"  IS 
CATHETER  IV  ANGIO_lS 
TAPE  SURGICAL  ADH  4S 
BANDAGE  ELAS_4IN_10S 
ADHE SI VE  TAPE  1  IN  1 2S 
GLOVE  LT  PF  LG_100S 
ADHES_TAPE  SILK  2IN  6S 
NEEDLE  HYP018GX1.5  100 
BANDAGE  ELAS_3IN  10S 
SPHYGMOMANOMETER  AD  IS 
SYRINGE  HYPO_5CC_100S 
ADHESIVE  SKIN  DERM  12S 
SYRINGE  HYPO_20CC_40S 
SYRINGE  HYPO_10CC_100S 

GLOVES_SURG_7.5 _ 1 00S 

SPONGE  4X4  TRAY  1280S 
BANDGE  COBAN  5X1  30S 
PAD  PREP  ALC_MED  200S 
BANDAGE  37X37X52IN  IS 


1 

2 

3 

4 

5 

6 

1474 

1412 

1103 

1121 

2486 

1834 

198 

294 

432 

762 

1672 

593 

3 

13 

5 

18 

131 

424 

181 

386 

347 

669 

339 

369 

527 

492 

611 

433 

420 

485 

424 

355 

424 

310 

760 

540 

446 

580 

1080 

833 

364 

382 

325 

204 

282 

270 

435 

222 

199 

290 

525 

428 

497 

136 

266 

264 

310 

246 

493 

97 

119 

158 

119 

234 

31 

146 

167 

92 

139 

105 

41 

18 

43 

47 

99 

70 

77 

113 

106 

39 

169 

88 

93 

119 

176 

158 

220 

150 

33 

73 

58 

58 

107 

75 

441 

76 

71 

73 

30 

52 

134 

84 

129 

59 

53 

53 

221 

264 

317 

664 

1062 

1510 

8599 

9898 

3226 

3065 

2988 

9212 

7 

8 

9 

10 

11 

12 

1706 

2327 

1405 

3170 

1845 

2410 

662 

343 

9224 

19309 

3242 

5017 

33 

37 

33 

243 

19 

18017 

325 

607 

399 

482 

317 

427 

723 

584 

679 

432 

550 

335 

471 

441 

425 

375 

380 

357 

1193 

1300 

869 

1119 

750 

1113 

244 

430 

399 

328 

345 

419 

329 

366 

328 

106 

367 

280 

287 

319 

292 

185 

280 

391 

119 

99 

120 

104 

218 

55 

341 

58 

179 

121 

120 

116 

115 

88 

100 

9 

79 

57 

118 

101 

136 

102 

165 

124 

211 

144 

173 

140 

98 

140 

118 

104 

112 

38 

176 

90 

331 

100 

703 

52 

74 

107 

60 

79 

237 

113 

66 

66 

563 

107 

575 

813 

392 

240 

10904 

9834 

3783 

970 

6445 

6936 

Table  23.  From/To  Travel  Time  Matrix  for  Supply  Chain  Locations  (Sites) 


Owens/Minor  Cardinal  Germany  Qatar 

Afghanistan  Hosp 

Balad  Hosp  Kirkuk  FDT  Iraq/MLB 

Kuwait  Hosp 

Tallil  FDT 

Tikrit  FDT 

Mosul  FDT 

Owens/Minor 

0 

6.55 

37.68 

52 

67.48 

60.05 

60.64 

61.78 

65.21 

64.02 

61.05 

59.72 

Cardinal 

6.55 

0 

43.28 

45 

69.79 

63.88 

63.1 

64.37 

67.79 

66.5 

64.02 

62.21 

Germany 

37.68 

43.28 

0 

12 

31.89 

21.4 

20.55 

21.56 

24.97 

23.02 

21.2 

19.61 

Qatar 

52 

45 

12 

0 

12.31 

8 

8.23 

7.01 

17.57 

17.88 

17.98 

19.13 

Afghanistan  Hosp 

67.48 

69.79 

31.89 

12.31 

0 

14.1 

14.02 

14.25 

12.91 

13.5 

14.16 

14.7 

Balad  Hosp 

60.05 

63.88 

21.4 

8 

14.1 

0 

2.54 

0.62 

3.16 

3.5 

0.86 

1.95 

Kirkuk  FDT 

60.64 

63.1 

20.55 

8.23 

14.02 

2.54 

0 

1.54 

4.72 

3.1 

0.68 

0.94 

Iraq/MLB 

61.78 

64.37 

21.56 

7.01 

14.25 

0.62 

1.54 

0 

3.44 

2.3 

1.23 

2.25 

Kuwait  Hosp 

65.21 

67.79 

24.97 

17.57 

12.91 

3.16 

4.72 

3.44 

0 

1.56 

4.6 

5.59 

Tallil  FDT 

64.02 

66.5 

23.02 

17.88 

13.5 

3.5 

3.1 

2.3 

1.56 

0 

3.1 

4.2 

Tikrit  FDT 

61.05 

64.02 

21.2 

17.98 

14.16 

0.86 

0.68 

1.23 

4.6 

3.1 

0 

1.15 

Mosul  FDT 

59.72 

62.21 

19.61 

19.13 

14.7 

1.95 

0.94 

2.25 

5.59 

4.2 

1.15 

0 

Given  these  model  inputs,  our  next  step  is  to  analyze  this  example  problem  instance  using 
our  optimization  model.  Our  mixed-integer  program  is  coded  in  AMPL  and  analyzed  using 
CPLEX  vlO.l.  The  test  case  scenarios  are  run  on  a  quad  core,  quad  processor  PC-based  server 
with  128  GB  of  RAM.  All  runs  produced  feasible  results  in  a  fraction  of  a  computational  second, 
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as  this  problem  falls  into  the  class  of  assignment  problems  which  can  be  resolved  quite  nicely 
using  standard  optimization  software  packages. 


Table  24.  Initial  Inventory  for  Top  20  Class  VIII  Lines  by  Prime  Vendor 


Line  Owens/Minor  Cardinal 


1 

13375 

13375 

2 

25048 

25048 

3 

11052 

11052 

4 

2817 

2817 

5 

3644 

3644 

6 

2920 

2920 

7 

6349 

6349 

8 

2395 

2395 

9 

2325 

2325 

10 

2083 

2083 

11 

1161 

1161 

12 

969 

969 

13 

459 

459 

14 

802 

802 

15 

1093 

1093 

16 

625 

625 

17 

1266 

1266 

18 

679 

679 

19 

4036 

4036 

20 

45516 

45516 

Table  25.  Assumed  Maximum  Total  Inventory  by  Site 


Owens/Minor 

15000 

Cardinal 

15000 

Germany 

8800 

Qatar 

8500 

Afghanistan  Hosp 

6844 

Balad  Hosp 

6878 

Kirkuk  FDT 

6779 

Iraq/MLB 

6980 

Kuwait  Hosp 

6569 

Tallil  FDT 

6556 

Tikrit  FDT 

6628 

Mosul  FDT 

6619 
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Figure  9  displays  the  model’s  transportation  cost-optimal  solution  for  ID  1  of  the  20 
Class  VIII  lines,  ‘Pad  Prep  Alcohol  100S.’  The  values  on  each  arc  indicate  the  quantity  of  the 
item  to  be  shipped.  In  order  to  both  meet  some  period  one  demand  levels  and  to  supply  network 
locations  with  inventory  for  subsequent  periods,  the  optimization  model  recommends  a  large 
initial  shipment  from  prime  vendor  Owens  &  Minor  to  Germany  (USAMMC-E),  which  is 
subsequently  sent  both  to  Qatar  and  Mosul.  The  Qatar  shipment  is  then  destined  for  other  supply 
locations  as  described  in  Figure  9. 

Notice  that  the  optimal  model  solution  in  Figure  9  determines  that  shipment  from  Qatar 
to  Iraq/MLB  to  the  Balad  hospital  is  preferred  to  a  direct  shipment  from  Qatar  to  the  Balad 
hospital.  This  is  most  probably  related  to  transportation  economies  of  scale  while  considering 
holding  costs  and/or  demand  levels  at  each  site.  By  allowing  this  sort  of  lateral  transshipment 
within  the  same  echelon  of  the  supply  chain,  our  optimization  model  is  able  to  exploit  all 
possible  transportation  and  distribution  options  in  order  to  find  a  globally  optimal  solutions  for 
the  Class  VIII  transportation  distribution  problem. 


Figure  9.  Time  Period  1  Model  Solution  for  Line  1  'Pad  Prep  Alcohol  100S' 
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Building  upon  the  example  model  results  displayed  in  Figure  9,  Table  26  displays  the 
resulting  projected  total  inventory  levels  by  site  by  month  over  the  model  horizon.  The  effects  of 
both  initial  deliveries  and  ongoing  demand  consumption  of  items  can  be  seen  in  the  figure  in 
terms  of  increasing  and  decreasing  inventory  levels,  respectively. 

Table  26.  Projected  Periodic  Inventory  Levels  for  Line  1  ‘Pad  Prep  Alcohol  100S’  by  Site 

Month 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

Owens/Minor 

5107 

4148 

3234 

2500 

2500 

2188 

1291 

1291 

1263 

724 

410 

0 

Cardinal 

13375 

13375 

13375 

12755 

10583 

8785 

5782 

4851 

4289 

3021 

0 

0 

Germany 

0 

0 

0 

50 

0 

0 

607 

0 

0 

0 

0 

0 

Qatar 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1157 

0 

Afghanistan  Hosp 

1158 

1059 

982 

904 

730 

602 

483 

320 

222 

0 

0 

0 

Balad  Hosp 

966 

825 

715 

603 

354 

171 

0 

0 

0 

0 

241 

0 

Kirkuk  FDT 

0 

0 

0 

0 

0 

0 

1338 

1059 

890 

510 

289 

0 

Iraq/MLB 

6980 

6268 

6186 

6041 

5724 

2659 

2599 

2568 

1738 

0 

0 

0 

Kuwait  Hosp 

0 

0 

0 

0 

202 

0 

760 

504 

349 

0 

0 

0 

Tallil  FDT 

0 

0 

0 

0 

0 

1120 

898 

595 

412 

0 

313 

0 

Tikrit  FDT 

0 

0 

0 

0 

330 

0 

0 

0 

0 

0 

0 

0 

Mosul  FDT 

212 

0 

0 

373 

0 

0 

0 

211 

0 

0 

0 

0 

5.6  Conclusions  and  Future  Research 

In  this  project,  we  developed  a  multi-commodity  flow  model  which  captures  many 
characteristics  of  network  design  problems.  The  main  characteristics  which  we  consider  include 
a  planning  horizon,  inventory  holding  and  initial  storages,  storage  and  commodity  flow 
restrictions  and  the  supply  of  commodities  between  any  identified  facilities  (production  plants, 
warehouses,  distribution  centers,  etc.).  As  all  organizations  operate  in  an  atmosphere  of 
uncertainty  and  decisions  must  be  made  that  affect  the  future  of  the  organization,  our  proposed 
optimization-based  transportation  and  distribution  approach  for  Class  VIII  materiel  can  be  a 
viable  way  for  military  decision  makers  to  act/react  to  changing  network  conditions. 
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In  the  future,  the  results  of  our  optimization  model  may  be  used  to  help  set  target 
inventory  levels  by  Class  VIII  item  type  and/or  by  site  such  that  inventory  replenishment  can  be 
done  more  cost  effectively.  Further,  anticipatory  actions  for  dealing  with  potential  network  node 
changes/closures  can  be  evaluated  using  this  optimization  framework  as  the  model  can  easily  be 
solved  in  a  ‘what  if  scenario  mode  by  simply  changing  the  AMPL  data  files  to  represent  the 
desired  network  configuration  under  study.  The  transportation  matrix  should  be  reevaluated  and 
updated,  with  a  large  (“big  M”)  value  being  input  for  any  transportation  routes/links  that  do  not 
exist  in  practice.  Finally,  additional  granularity  into  the  demand  resolution  at  each  node  in  each 
period  could  be  added  to  allow  the  model  to  solve  more  realistic  problem  instances,  rather  than 
the  example  instance  the  was  based  on  an  abstraction  of  actual  demand  values  for  a  whole  year. 
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6.1  Executive  Summary 

This  task  aimed  to  increase  the  understanding  and  knowledge  of  disaster  relief  management 
within  the  contexts  of  managing  the  limited  resources.  Specifically,  the  problem  addressed  in  this 
research  is  the  inefficiency  in  resource  allocation  for  disaster  relief  procurement  operations.  The 
research  activities  focused  on  understanding  the  resource  allocation  problems  considering 
different  dimensions.  A  holistic,  intuitive,  and  reconfigurable  procurement  auctions-based 
framework  for  effective  resource  allocation  was  developed  in  which  auctioneers  and  bidders 
compete  amongst  each  other  in  multiple  rounds  of  the  procurement  auction.  In  decentralized  and 
chaotic  disaster  relief  environment  the  developed  model  offers  a  systematic  approach  to  allocate 
resources  effectively.  In  this  alternative  model  unique  system  parameters  were  introduced  and 
tested  in  order  to  utilize  all  available  resources  on-hand.  These  parameters  include  substituting 
the  required/demanded  items  by  other  on-hand  items  and  allowing  partially  fulfilled  demands 
(i.e.  allowing  the  amount  of  shipped  item  being  less  than  required/demanded)  to  satisfy  a  portion 
of  them.  Moreover,  procedures  to  handle  urgency  of  the  demands  and  sorting  them  were 
addressed  in  the  framework.  Using  these  system  characteristics  of  a  disaster  relief  environment, 
the  framework  is  verified  by  simulation  and  optimization  techniques. 
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6.2  Introduction 


The  occurrence  of  natural  disasters  is  increasing  especially  in  the  last  two  decades.  225 
natural  disasters  were  recorded  on  a  yearly  average  between  1987  and  1997,  whereas  the  same 
figure  almost  doubled  between  2000  and  2006  [8,13].  This  number  reached  to  414  in  2007  [13]. 
The  property  damage  from  Hurricane  Katrina  in  2005  alone  was  estimated  about  $96  billion;  the 
residential  damage  was  approximately  300,000  homes  [16].  Thousands  were  left  injured  or 
missing  in  Indonesia,  India,  Sri  Lanka  and  Thailand  after  tsunamis  in  2004  and  the  earthquake  in 
China  this  year.  It  was  very  difficult  or  impossible  for  many  homeless  victims  to  reconstruct  their 
shattered  lives.  The  crucial  questions  are  how  responsive  and  prepared  we  are  to  these  types  of 
tragedies  and  how  we  respond  when  they  occur. 

Disaster  relief  management  domain  is  distributed  and  has  a  decentralized  structure  due  to:  (1) 
the  domain  is  geographically  distributed,  (2)  relief  efforts  typically  require  separate  independent 
organizations,  and  (2)  each  organization  has  unique  business  processes  to  perfonn  particular 
tasks  [17].  Organizations  (e.g.,  local  governments,  the  United  Nations,  and  non-governmental 
organizations  etc.)  in  this  environment  reveal  autonomous  characteristics  in  order  to  achieve 
their  own  goals.  They  are  usually  self-centered  with  specific  decision-making  mechanisms  and 
expertise  and  often  times  they  are  reluctant  to  release  their  process  information.  They  typically 
have  less  formalized  internal  and  external  information  and  communication  systems.  Thus,  the 
knowhow  gained  by  one  organization  cannot  be  transferred  to  others  easily.  Moreover,  every 
disaster  relief  operation  is  treated  in  unique  ways  and  there  is  no  standard  means  of  operations 
used  by  those  organizations. 
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6.3  Methods,  Assumptions,  and  Procedures 

Given  that  local  resources  are  vital  in  the  first  few  days  after  a  disaster  strikes,  they  should  be 
utilized  efficiently  to  supply  the  needs  of  the  victims  immediately.  Additionally,  procurement 
activities  should  be  performed  according  to  the  specifics  of  disaster  relief  operations  in  the  long 
run.  The  framework  proposed  for  disaster  relief  operations  here  fits  well  both  into  the  immediate 
response  with  local  resources  and  also  the  long-tenn  procurement  activities  from  local  and  global 
suppliers.  The  main  idea  is  to  introduce  some  auction  design  parameters  and  decision  making 
logic  that  would  facilitate  the  procurement  activities. 

The  procurement  auctions  considered  in  this  research  have  multiple  auctioneers  and  multiple 
bidders.  The  bidder  parties  can  be  identified  as  warehouses,  suppliers,  or  manufacturers  of  the 
auctioned  items  in  a  disaster  relief  environment,  whereas  the  auctioneer  parties  represent  the 
disaster  locations  which  send  appeals  for  the  items.  The  auction  process  is  given  in  Figure  10. 
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Figure  10.  Procurement  Auctions  Framework 

As  seen  in  Figure  10,  in  the  announcement  construction  phase,  the  auctioneer  accumulates 
incoming  demands  and  releases  announcements  based-on  a  predefined  count  threshold  for 
priority  of  items  (i.e.,  a  certain  number  of  highest  priority  items).  Demands  for  different  items 
arrive  to  auctioneers  as  appeals  for  relief  supplies.  These  demands  are  then  bundled  according  to 
the  announcement  construction  criterion.  When  there  are  enough  demands  to  form  an 
announcement,  it  is  announced.  The  auctioneer  accumulates  demands  to  benefit  from  the 
economies  of  scale  in  the  procurement  process.  The  bid  construction  phase  receives  an 
announcement,  compares  demanded  items  with  on  hand  inventory  quantities,  and  their 
associated  values.  The  value  of  an  item  corresponds  to  its  age  and  condition.  Using  this 
information,  bidders  (suppliers)  determine  the  quantities  and  mix  of  their  bids  with  the  aim  of 
minimizing  the  current  asset  value  of  offered  items.  In  the  bid  evaluation  phase,  bid  quantities 
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and  their  associated  asset  values  are  maximized  by  a  general  multidimensional  knapsack  problem 
(MDKP).  Details  of  the  framework  will  be  explained  in  separate  sections.  The  following 
considerations  are  made  in  the  auction  process: 

•  When  an  auction  is  finalized,  the  procured  items  from  the  bidder  to  the  auctioneer  are 
considered  to  be  shipped  and  consumed. 

•  Without  loss  of  generality,  there  is  only  one  substitute  for  each  item  and  two  or  more  order 
substitutes  (i.e.,  substitute  of  a  substitute)  are  not  allowed. 

•  An  announcement  cannot  have  the  original  item  and  the  substitute  item  at  the  same  time. 

The  auctioneer  is  responsible  for  setting  the  stage  for  the  auction.  An  auctioneer  can  offer 

two  announcement  options  in  the  announcement  construction  phase.  These  are  substitution  and 
partial  fulfilment  options.  These  options  are  proposed  in  order  to  fulfill  the  demand  of  the 
disaster  locations  as  much  as  it  is  possible  with  the  current  inventories  of  bidders.  If  the 
substitution  option  is  preferred  by  the  disaster  locations,  then  these  unsolicited  donations  will  get 
a  chance  to  be  used  instead  of  causing  stocking  costs.  Substitute  item  options  are  given  to  the 
bidder  to  give  the  opportunity  to  bid  on  the  item  even  if  it  does  not  have  the  original  quantity. 
The  partial  fulfilment  option  enables  better  usage  of  supplier  inventories,  and  the  value  of  the 
item  gives  a  means  to  humanitarian  organizations  to  evaluate  the  supplies. 

The  priority  of  items  is  included  in  the  framework  to  improve  the  linkage  between  the 
disaster  locations  and  suppliers.  Three  levels  of  priority  are  used  in  the  proposed  framework.  The 
first  level  indicates  urgent-immediate  distribution,  the  second  level  indicates  low  priority 
distribution,  and  the  third  level  indicates  non-priority  items.  The  ease  of  logistics  concept 
attempts  to  take  into  account  of  the  differences  among  suppliers  in  terms  of  convenience  in 
geographical  or  topographical  access  to  the  disaster  location.  During  disasters,  essential 
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infrastructure  like  highways,  roads  and  bridges  are  usually  destroyed.  The  ease  of  logistics 
parameter  is  considered  in  three  levels.  The  suppliers  having  better  ease  of  logistics  are  favoured 
in  the  bid  evaluation. 

6.3.1  Announcement  construction  phase 

In  the  announcement  construction  phase,  appeals  for  items  are  declared  with  item  types, 
quantities  and  priorities.  Each  item  type  has  a  priority,  an  integer  from  the  interval  [1,3]  (with 
one  being  the  highest  priority).  When  a  certain  quantity  threshold  is  reached  for  the  highest 
priority  items,  the  decision  for  partial  fulfilment  and  substitution  options  is  taken  and  the 
announcement  is  constructed.  Time-to-fill  for  the  announcement  is  defined  as  the  waiting  time  of 
the  announcement  before  receiving  any  response  from  bidders.  An  upper  bound  for  time-to-fill 
was  selected  as  24  time  units  (e.g.,  hours,  etc.),  because  the  assessment  happens  within  the  first 
24  hours  after  the  disaster  strikes  [15].  The  weighted  priority  ( WP )  of  the  announcement  is  used 
in  extrapolating  from  [1,3]  priority  interval  to  [1,24]  time  unit  interval  using  [ time  to  fill  = 
1  +  (weighted  priority  -  1)  x  23/2].  Note  that  higher  priority  announcements  have  a  shorter 
time-to-fill. 

6.3.2  Bid  construction  phase 

In  the  bid  construction  phase,  bidders  need  to  decide  whether  to  bid  or  not  on  the  announcement. 
This  decision  is  based  on  the  urgency  of  the  announcement  and  the  bidding  strategy  that  the 
bidder  follows.  If  the  announcement  is  urgent,  the  bidder  by-passes  its  bidding  strategy  and 
constructs  bid  for  the  announcement.  Otherwise,  it  checks  whether  its  bidding  strategy  picks  the 
upcoming  announcement.  If  the  announcement  is  chosen,  then  the  bidder  constructs  the  bid; 
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otherwise  the  bidder  notifies  the  auctioneer  with  a  null  bid.  Bidders  make  the  comparison  among 
announcements  with  three  different  strategies: 

1 .  Bid  to  the  announcement  if  it  has  the  longest  waiting  time  in  the  announcement  queue. 

2.  Bid  to  the  announcement  if  it  has  the  highest  fill  rate  (i.e.,  supplied  amount  /  requested 
amount)  with  original  item  types. 

3.  Bid  to  the  announcement  if  it  is  the  most  urgent  (i.e.,  the  lowest  weighted  priority). 

These  strategies  are  applied  only  when  there  are  enough  announcements  (three 
announcements  are  taken  without  loss  of  generality)  in  bidder’s  agenda  to  compare.  The  first 
strategy  aims  to  decrease  the  waiting  time  of  announcements  in  the  queue.  The  second  strategy 
aims  to  better  utilize  on  hand  inventories  and  the  third  strategy  gives  importance  to  the  priority  of 
the  items. 

After  a  bidder  decides  to  bid  on  an  announcement,  it  uses  an  IP  formulation  to  construct  its 
bid.  The  decision  in  bid  construction  is  whether  to  use  substitute  items  or  not  while  fulfilling  the 
announcement  with  original  items.  A  bidder  may  have  choices  of  satisfying  the  demand  with 
only  original  items,  only  substitute  items,  or  a  mix  of  those  depending  on  its  inventory  on  hand. 
The  objective  function  used  in  bid  construction  is  formulated  as:  Vj  +  YjWj),  where  Xj  is 

the  original  quantity  bid,  Yj  is  the  substitute  quantity  bid,  Vj  and  Wj  are  the  original  and  substitute 
values  of  the  bidders’  inventory  for  item  j  of  the  announcement  having  m  items.  The  value  of  an 
item  represents  the  asset  value  of  the  item  depending  its  age  and  condition.  The  challenge  here  is 
whether  to  include  substitutes  and  how  much  to  include  when  it  is  allowed  by  the  auctioneer 
party.  The  bids  are  divisible  and  all-or-nothing  bids  are  not  accepted,  therefore  suppliers  are 
considered  as  willing  to  give  the  quantity  that  is  allocated  by  the  auctioneer  at  the  same  value  as 
they  offered  for  the  whole  quantity  [14,18].  In  the  following  formulation,  if-then  constraints  and 
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inventory  availability  parameters  are  critical.  If-then  constraints  are  needed  for  the  partial 
demand  fulfilment  and  the  substitution  options.  The  inventory  availability  parameter  aims  to 
determine  the  inventory  on  hand.  The  index  of  items  in  an  announcement  in  the  IP  fonnulation  is 
represented  by  (j  =  I The  parameters  and  decision  variables  are  given  in  Table  27. 

Table  27  Parameters  and  decision  variables  in  bid  construction  phase 


Parameter 

Definition 

Qi 

original  demand  quantity  for  item  type  j 

ij 

original  quantity  of  type  j  in  bidder’s  inventory 

Hj 

substitute  quantity  of  type  j  in  bidder’s  inventory 

Vj 

value  of  original  type  j  in  bidder’s  inventory 

Wj 

value  of  substitute  type  j  in  bidder’s  inventory 

Pj 

If  if  partialdemandfulfdlmetisallowedfor  type/ 
[0,  otherwise 

Sj 

f  1,  if  substitute  type  is  allowed  for  type  j 
[O,  otherwise 

ZJ 

[0,  if  inventoryof  bidder  is  greater  than  announcedquantity  for  type  j 
[1,  otherwise 

M 

Big-M  (i.e.  a  sufficiently  large  integer) 

Decision  Variables 

% 

original  quantity  bid  by  the  retailer 

Y, 

substitute  quantity  bid  by  the  retailer 

The  objective  function  is  given  as  [ Min  Y’J'i.Xj  P/  +  YjWj)].  The  constraints  of  the  IP 
formulation  are  as  follows: 


Xj  +  SjYj>  Qj-Mzj 

Vj 

Xj  —  Pj  7/  -  M(  1  -  zj) 

Vj 

Yj<MSj 

Vj 

Yj^SjPjHj-Md-zj) 

Vj 

Xj<Ij 

Vj 

Xj>  0  and  integer 

Vj 

Yj<Hj 

Vj 

Yj  >  0  and  integer 

Vj 

6.3.3  Bid  evaluation  phase 

In  the  bid  evaluation  phase,  the  auctioneer  collects  responses  from  bidders  and  decides  whether 
or  not  to  send  the  announcement  back  to  bidders  for  another  round.  Multi-round  auctioning 
usually  means  the  revision  of  the  bid  from  suppliers  [3];  on  the  other  hand,  the  framework  in  this 
research  introduces  modification  of  announcements  to  get  a  higher  fill  rate  in  the  upcoming 
rounds.  In  our  framework,  the  willing-to-give  ratio  is  important.  The  willing-to-give  ratio  is 
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defined  as  [(total  bid  quantity)  /  (announcement  quantity)].  This  ratio  is  calculated  for  each  item 
using  all  the  bids.  When  there  are  enough  bids  to  reach  the  predetermined  willing-to-give  ratio 
for  all  item  types,  then  auctioneer  evaluates  the  bids.  If  there  are  not  enough  bids,  then  the 
priority  of  the  announcement  is  increased  by  a  priority  increase  rate,  time-to-fill  is  updated  (i.e., 
decreased),  and  the  announcement  is  sent  back  to  the  bidders  for  a  second  round.  In  the  second 
round,  if  the  willing-to-give  ratio  is  still  not  reached,  then  the  substitution  and  partial  fulfilment 
options  are  turned  on  for  the  item  types  where  they  were  not  allowed  before.  Then,  the  priority 
of  the  announcement  is  increased,  time-to-fill  is  updated,  and  the  announcement  is  sent  back  to 
the  bidders  for  a  third  round.  If  the  willing-to-give  ratio  is  still  not  reached,  then  the  auctioneer 
evaluates  the  bids  and  becomes  content  with  the  available  bids. 

The  bid  evaluation  phase  chooses  the  suppliers  to  fulfil  the  announcement.  The  auctioneer 
might  fulfil  the  announcement  by  only  original  items,  only  substitute  items,  or  a  mix  of  those 
depending  on  the  bids  received  and  the  location  of  the  bidders.  The  objective  function  used  in 
bid  evaluation  is  formulated  as:  VLJ  +  fi(/TT(J),  where  Ay  is  the  original  quantity  of 

item  j  allocated  to  bidder  i,  By  is  the  substitute  quantity  of  item  j  allocated  to  bidder  i,  and  Vy  and 
Wy  are  the  original  and  substitute  values  of  the  bidder  V s  inventory  for  item  j  in  an 
announcement  having  m  items.  Note  that  Vy  and  Wy  are  exogenous  for  the  auctioneer  and 
declared  by  the  bidder  in  the  bid  construction  phase.  Here,  cq  represents  the  ease  of  logistics 
parameter  for  bidder  i.  The  following  formulation  is  a  variation  of  the  general  MDKP  [1].  The 
index  of  items  in  an  announcement  in  the  IP  fonnulation  is  represented  by  (j  =  1  and  the 

index  of  bidders  is  represented  by  (i  =  1 ,...,«). 

The  objective  function  is  given  as  [Max  VLy  -I-  BijWij')]. The  constraints  of  the 

IP  formulation  are  as  follows: 
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v; 


Ajj>  0  and  integer 


n 

+  Bij)  -  Qj 

Aij<Cij  Vi,j 

B‘j  —Dy  Vi,; 


Vi,;' 

By  -  0  and  integer  Vi,; 


<5.3.4  Experimental  Study 

Proposed  procurement  auctions-based  framework  is  evaluated  with  respect  to  different 
design  parameters.  A  simulation  model  was  used  to  generate  different  problem  instances  and  to 
combine  the  three  phases  of  the  framework.  The  results  of  two  sets  of  experiments  are  analyzed 
and  presented  here:  the  first  set  of  experiments  illustrates  the  effects  of  the  environmental  factors 
and  the  second  set  of  experiments  details  the  effects  of  different  auction  parameters  with  four 
special  scenarios.  The  environmental  and  auction  design  parameters  are  depicted  in  Figure  11. 

In  Figure  11,  environmental  factors  are  given  as  (1)  demand  quantity  distribution,  (2)  the 
ease  of  logistics,  (3)  value  of  the  item,  (4)  inventory  on  hand,  and  (5)  lead  time.  Auctioneer 
related  parameters  are  given  as  willing-to-give  ratio,  and  strategy  threshold.  The  count  threshold, 
priority  increase  rate,  and  bidder  related  parameter  are  given  for  the  bidding  strategy.  In  the 
following  two  sections  environmental  factors  and  auction  design  parameters  are  explained. 


Figure  10.  Experimental  design  factors 
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6.3.5  Environmental  factors 

The  location  and  timing  of  a  disaster  determine  some  environmental  factors  which  can  be 
changed  neither  by  the  auctioneers  nor  by  the  bidders.  These  environmental  factors  affect  the 
result  of  the  procurement  auctions,  but  are  not  an  inherent  part  of  the  auction  design.  In  an 
attempt  to  stabilize  the  environmental  factors  for  the  scenario  analysis,  a  25  full  factorial  design 
of  experiments  was  performed  with  three  bidders  and  one  disaster  location.  One  high  and  one 
low  level  was  chosen  from  Table  28.  Demands  for  different  item  types  were  generated  using  a 
Poisson  distribution  with  a  mean  of  1  demand/time-unit.  Each  item  type  is  equally  likely  to  be 
demanded  out  of  ten  item  types.  One-thousand  individual  demands  are  generated  with  30 
replications.  The  quantity  of  each  demand  is  a  random  variable  which  follows  the  demand 
quantity  distribution.  Each  incoming  demand  quantity  is  added  to  its  same  item  type  in  the 
bundle  and  the  count  threshold  (200)  is  checked.  If  the  threshold  is  met,  then  this  bundle,  which 
includes  quantities  for  different  item  types,  is  announced.  The  demand  quantity  distribution 
depends  on  the  severity  of  the  damage  in  the  disaster  location.  Ease  of  logistics  is  a  constant 
factor  relative  to  disaster  locations  and  bidders,  which  is  detennined  after  the  disaster  strikes. 
Value  of  the  item,  inventory  on  hand,  and  lead  time  are  determined  by  contractual  terms  between 
external  lead  time  demand  fillers  and  bidders.  The  S  value  represents  the  order  up  to  level  and 
the  s  value  represents  the  reorder  point,  which  is  taken  as  ten  for  all  bidders. 


Table  288.  Environmental  Factors  and  Their  Levels 


Factor 

Level 

Low 

High 

Demand  quantity  distribution 

U(l,50) 

U(100,150) 

Ease  of  logistics 

33%  of  the  bidders  -  3 

67%  of  the  bidders  -  1 

33%  of  the  bidders  -  1 

67%  of  the  bidders  -  3 

Value  of  the  item 

33%  of  the  bidders  -  100 
67%  of  the  bidders  -  50 

33%  of  the  bidders  -  50 

67%  of  the  bidders  -  100 
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Inventory  on  hand 

33%  of  the  bidders  -  70 

67%  of  the  bidders  -  30 

33%  of  the  bidders  -  30 

67%  of  the  bidders  -  70 

Lead  Time 

33%  of  the  bidders  -  U(12,  72) 
67%  of  the  bidders  -  U(72,120) 

33%  of  the  bidders  -  U(72,  120) 
67%  of  the  bidders  -  U(  12,72) 

One  of  the  performance  measures  in  humanitarian  logistics  is  the  quantity  that  is  supplied  out 
of  the  amount  requested  [5],  This  metric  is  defined  as  the  fill  rate.  A  practical  example  for  the  fill 
rate  can  be  given  from  a  disaster  relief  operation  after  South  Asia  Earthquake  on  October  9th, 
2005.  The  fill  rate  (i.e.,  appeal  coverage)  was  63%  after  the  first  week,  47%  after  the  second 
week,  74%  after  a  month,  91%  after  two  months,  and  93%  after  three  months  [5].  Allocation 
share  of  bidders  is  another  perfonnance  measure,  which  defines  the  distribution  of  supplied 
items  among  bidders.  When  the  results  of  the  full  factorial  design  are  analyzed,  the  value  of  the 
item  and  ease  of  logistics  factors  have  no  effect  on  the  fill  rate.  Other  quantity  related  factors 
have  an  effect  on  fill  rate.  Inventory  on  hand  and  ease  of  logistics  affect  the  bidder  shares,  where 
higher  inventory  on  hand  and  higher  ease  of  logistics  increase  the  bidder’s  share.  For  the 
scenario  analysis,  middle  levels  from  Table  28  were  selected  and  remained  unchanged  during  the 
remaining  experiments.  Demand  quantity  distribution  was  selected  as  U(50,100)  for  all  disaster 
locations.  The  value  of  the  item  in  a  bidders’  inventory  is  taken  as  75,  inventory  on  hand  was  set 
to  100  and  ease  of  logistics  was  set  to  2  for  all  bidders.  The  fill  rate  of  the  auctioneer  with  these 
middle  levels  is  0.59. 

6.3.6  Auction  design  parameters 

Auctioneer  related  parameters  give  flexibility  to  the  auctioneers  to  adapt  themselves  to  the 
changing  settings  of  different  disasters  and  different  locations.  Willing-to-give  ratio  detennines 
the  eagerness  of  the  disaster  location  for  demand  satisfaction.  Since  resources  are  limited  in  a 
disaster  relief  environment,  there  might  be  cases  where  auctioneers  need  to  be  satisfied  with  a 
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certain  amount.  This  concept  is  tenned  willing-to-give  because  it  shows  the  willingness  of  the 
bidders  to  bid  on  an  announcement.  The  strategy  threshold  is  a  safety  factor  for  disaster  locations 
where  the  below  threshold  levels  by-pass  bidder  strategies  in  urgent  announcements  and  oblige 
bidders  to  bid  on  the  announcement.  Count  threshold  determines  the  number  of  highest  priority 
items  and  the  timing  of  an  announcement.  Priority  increase  rate  detennines  the  rate  to  increase 
the  priority  between  multiple  rounds.  We  assume  that  the  tolerance  of  the  disaster  location 
decreases  with  increasing  rounds.  Bidding  strategy  gives  flexibility  to  bidders  whether  to  bid  or 
not.  Table  29  summarizes  the  special  scenarios  to  analyze  these  auction  parameters. 


Table  29.  Four  scenarios  to  evaluate  different  auction  design  parameters. 


Scenario 

Bidders 

Auctioneers 

Performance 

measure 

Target  design 
parameter 

1 

-  No  decision  on 
whether  to  bid  or  not 

-  Construct  bid  to 
every  announcement 

-  Evaluate  all  the  bids. 

-  Single  Round 

-  fill  rate 

-  announcement 

queue 

-  Base  case  for 
multiple  disaster 
locations 

-  Count  threshold 

2 

-Decide  to  bid  or 
not 

-3  bidders-strategy  1 
-3  bidders-strategy  2 
-3  bidders-strategy  3 

-  Evaluate  all  the  bids. 

-  Single  Round 

-  Count  threshold  same 
for  all  auctioneers  (400) 

-  fill  rate 

-  allocation  share 
among  bidders 
-announcement 

queue 

-  Effect  of  bidder 
decision  making  and 
bidding  strategies 

-  Strategy  threshold 

3 

-  No  decision  on 
whether  to  bid  or  not 

-  Construct  bid  to 
every  announcement 

-Decide  to  evaluate  bids 
-Multiple  rounds 
-Count  threshold  same 
for  all  (400) 

-Strategy  threshold 
same  for  all  ( 1 .7) 

-  fill  rate 
-announcement 

queue 

-Effect  of  auctioneer 
decision  making 
-Effect  of  multiple 
rounds 

-Willing  to  give  ratio 

4 

-Decide  to  bid  or  not 
-3  bidders-strategy  1 
-3  bidders-strategy  2 
-3  bidders-strategy  3 

-Decide  to  evaluate  bids 
-Multiple  rounds 
-Count  threshold  same 
for  all  (400) 

-Strategy  threshold 
same  for  all  ( 1 .7) 
-Wiling-to-give  same 
for  all  (0.7) 

-  fill  rate 

-  allocation  share 
among  bidders 
-announcement 

queue 

-Effect  of  auctioneer 
decision  making  and 
bidder  decision 
making  combined 
-Effect  of  multiple 
rounds 

-Priority  increase 
rate 
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The  experiments  in  Table  29  are  perfonned  with  three  auctioneers  and  nine  bidders.  Only  the 
target  design  parameters  are  altered  in  each  scenario.  After  the  target  design  parameter  is 
analyzed,  it  remained  unchanged  in  later  scenarios.  The  first  scenario  is  the  base  case  for  the 
multiple  auctioneers  setting.  In  the  second  scenario,  bidding  strategies  are  introduced.  The  third 
scenario  introduces  multiple  rounds,  and  the  fourth  scenario  evaluates  bidding  strategies  and 
multiple  rounds  together.  The  fill  rate,  the  allocation  share  among  bidders  and  the  announcement 
queue  characteristics  are  used  as  the  performance  factors  [4].  An  announcement  queue  holds  all 
the  announcements  that  an  auctioneer  has  requested. 

The  objective  of  the  first  scenario  is  to  examine  the  effect  of  multiple  disaster  locations  and 
count  threshold  on  fill  rate.  Neither  bidders  nor  the  auctioneers  have  decision  making  logic; 
bidders  bid  to  every  announcement  and  auctioneers  get  whatever  they  are  given.  This  scenario  is 
like  a  single  round  sealed-bid  auction.  The  results  are  given  in  Table  30.  It  can  be  concluded  that 
total  number  of  announcements,  number  of  announcements  in  the  announcement  queue  and  fill 
rate  decreases  with  increasing  count  threshold  levels.  With  one  auctioneer  and  three  bidders,  the 
fill  rate  was  0.59  compared  to  the  0.71  fill  rate  with  three  auctioneers  and  nine  bidders  in 
experiment  one.  This  increase  can  be  by  explained  by  the  pooling  of  different  bidder  inventories. 
A  count  threshold  was  selected  as  400  for  future  scenarios  with  a  fill  rate  of  0.61. 


Table  300.  First  scenario  results 


Experiment 

Auctioneer 

Count 

threshold 

Fill 

rate 

Total 

Number  of 
Ann.s 

Average  Time 
an 

Announcement 
Spends  in 
Queue 

Average 
Number  of 
Announcement 
s  in  Queue 

1 

200 

0.71 

104.67 

11.20 

1.17 

1 

2 

200 

0.71 

104.67 

11.18 

1.16 

3 

200 

0.71 

104.67 

11.24 

1.17 

1 

200 

0.73 

104.67 

11.20 

1.17 
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2 

2 

3 

400 

600 

0.61 

0.51 

55.93 

38.37 

11.76 

12.06 

0.65 

0.46 

1 

400 

0.61 

56.63 

11.82 

0.67 

3 

2 

400 

0.61 

55.93 

11.76 

0.65 

3 

400 

0.61 

56.00 

11.82 

0.66 

The  second  scenario  examines  the  effect  of  bidder  decision  making,  bidder  strategies,  and 


strategy  threshold.  Three  out  of  nine  bidders  all  behave  according  to  one  bidding  strategy.  In  this 


analysis,  the  strategy  threshold  is  altered.  This  scenario  is  like  a  single  round  sealed-bid  auction 


with  increased  governance  on  the  bidder  side.  The  results  are  given  in  Table  31.  Urgency 


increases  by  decreasing  levels  of  weighted  priority  (WP)  and  bidder  strategies  are  by-passed  for 


an  urgent  announcement.  Since  E[WP]  =  2,  probabilistically  less  number  of  announcements  fall 


into  below  threshold  levels.  It  can  be  seen  that  lower  levels  for  strategy  threshold  give  bidders 
more  freedom  of  not  bidding,  therefore  fill  rate  decreases.  The  third  bidding  strategy  emphasizes 
announcements  with  higher  weighted  priority;  consequently  bidders  7-9  have  the  highest  shares 
in  the  allocation.  Bidders  using  the  second  bidding  strategy  outperform  the  bidders  with  the  first 
bidding  strategy,  because  the  second  bidding  strategy  picks  the  announcements  with  higher  fill 
rate.  The  announcement  queue  shows  similar  characteristics  in  this  scenario  when  compared  with 
the  first  scenario  having  the  400  count  threshold  level.  A  strategy  threshold  of  1 .7  was  selected 
for  the  last  two  scenarios  for  all  auctioneers. 


Table  311.  Second  scenario  results 


Experiment 

Auctioneer 

Strategy 

threshold 

Fill  rate 

Sum  of 
Bidders  1-3 
Share 

Sum  of 
Bidders  4-6 
Share 

Sum  of 
Bidders  7-9 
Share 

1 

1.5 

0.04 

1 

2 

1.5 

0.05 

0.07 

0.22 

0.71 

3 

1.5 

0.04 

1 

1.5 

0.04 

2 

2 

1.7 

0.12 

0.14 

0.26 

0.61 

3 

1.9 

0.30 
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1 

1.7 

0.10 

2 

1.7 

0.12 

0.12 

0.24 

0.64 

3 

1.7 

0.12 

The  third  scenario  examines  the  effect  of  multiple  rounds,  auctioneer  decision  making,  and 
willing-to-give  ratio.  In  this  scenario,  bidders  do  not  have  bidding  strategies.  During  multiple 
rounds  of  auctioning,  the  auctioneer  alters  the  parameters  of  the  announcement  to  reach  higher 
fill  rates.  In  each  round  when  the  willing-to-give  ratio  is  not  reached,  for  the  items  having  WP  > 
2,  priority  is  increased  by  the  priority  increase  rate.  For  instance,  an  item  with  2.2  WP  would 
have  2. 1  WP  after  one  round  of  auction  with  a  0. 1  priority  increase  rate.  In  the  third  scenario,  the 
priority  increase  rate  is  taken  as  0.1  for  all  auctioneers.  The  results  are  given  in  Table  32.  Fill 
rates  jumped  substantially  to  closer  values  to  1.0.  Since  the  willing-to-give  ratio  is  related  to 
bidder  strategies,  it  shows  a  slight  effect  on  fill  rates.  If  the  willing-to-give  ratio  is  increased,  fill 
rates  slightly  decrease.  Altering  the  willing-to-give  ratio  changes  the  number  of  announcements 
resolved  in  each  round.  If  the  willing-to-give  ratio  is  increased,  it  pushes  the  announcements 
back  to  the  later  rounds.  The  number  of  resolved  announcements  in  the  second  round  is  the  least, 
which  shows  that  announcement  options  do  not  change  the  decision  of  sending  the 
announcement  back  to  the  bidders.  The  average  time  that  an  announcement  spends  in  the  queue 
and  the  average  number  of  announcements  in  the  queue  almost  doubled.  It  can  be  concluded  that 
auctioneers  wait  more  in  order  to  reach  the  higher  fill  rates  in  later  rounds.  The  willing-to-give 
ratio  was  set  to  0.7  for  the  last  scenario. 
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The  fourth  scenario  examines  the  combined  effects  of  bidder  and  auctioneer  decision  making  in 
multiple  rounds  with  priority  increase  rate.  This  scenario  includes  all  the  design  parameters  that 
are  proposed.  Priority  increase  rate  works  together  with  strategy  threshold  to  by-pass  the  bidder 
strategies.  The  results  are  given  in  Table  33.  If  priority  is  increased  by  0.5  between  rounds,  then 
bidder  strategies  are  by  passed  and  bidders  are  obliged  to  bid,  which  leads  to  higher  fill  rates.  If 
priority  is  increased  by  0.1  between  rounds,  then  bidders  might  decide  not  to  bid,  which 
decreases  fill  rate.  If  the  priority  increase  rate  is  smaller,  more  announcements  are  pushed  to  later 
rounds  to  be  resolved.  When  auctioneers  use  different  priority  increase  rates  as  0.1,  0.1  and,  0.5, 
respectively,  the  third  auctioneer  with  0.5  does  not  reach  to  the  high  fill  rate  when  they  all  use 
0.5.  If  the  auctioneers  use  0.5,  0.5,  and  0.1,  respectively,  the  third  auctioneer  with  0.1  does  not 
have  a  fill  rate  as  low  as  when  they  all  use  0.1.  The  higher  priority  increase  rate  makes 
announcements  spend  less  time  in  the  queue  and  enable  higher  fill  rates  to  be  reached.  The 
bidder  shares  change  significantly.  Lower  priority  increase  rate  makes  first  biding  strategy  more 
powerful,  whereas,  bidders  with  the  second  bidding  strategy  get  more  shares  with  a  higher 
priority  increase  rate. 

In  order  to  compare  different  scenarios  in  terms  of  fill  rate,  the  experiments  on  which  the 
later  scenarios  built  were  selected.  The  third  experiment  from  the  first,  second,  and  third 
scenarios,  and  the  first  experiment  from  the  last  scenario  were  examined.  The  average  fill  rates 
of  auctioneers  and  the  announcement  queue  characteristics  are  shown  in  Table  34.  The  first 
scenario  reaches  a  0.61  fill  rate,  whereas  in  the  second  scenario,  fill  rate  decreases  substantially, 
because  most  of  the  bidders  choose  not  to  bid  and  auctioneers  do  not  have  a  means  to  compel 
them  to  bid.  These  two  scenarios  have  the  least  time  spent  and  number  in  announcement  queue. 
The  third  scenario  introduces  auctioneer  decision  making  in  multiple  rounds,  and  if  the 
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auctioneer  is  not  satisfied  with  what  has  been  bid,  it  sends  the  announcement  back  to  the  bidders. 
This  allows  more  time  for  the  bidders  to  replenish  their  inventory  and  gives  priority  to  the  second 
and  third  round  announcements  in  the  announcement  queue.  As  a  result,  announcements  spend 
more  time  in  the  queue,  but  reach  the  highest  fill  rate.  This  might  be  unrealistic,  because  bidders 
do  not  have  the  decision  to  bid  or  not.  In  the  fourth  scenario,  although  the  time  an  announcement 
spends  in  the  queue  increases,  it  reaches  a  higher  fill  rate  than  the  first  and  the  second  scenario. 

Table  32.  Third  Scenario  Results 


Exp. 

Willing- 
Auctioneer  to-give 
ratio 

Fill 

rate 

Average 
Time  an 
Ann. 
Spends 
in 

Queue 

Average 
Number 
of  Ann.s 
in 

Queue 

Total 

Number 

of 

Ann.s 

Number 

of 

Resolved 

Ann.s 

First 

Round 

Resolved 

Second 

Round 

Resolved 

Third 

Round 

Resolved 

1 

0.5 

0.9992 

22.63 

1.27 

56.63 

54.80 

9.87 

0.53 

44.40 

1 

2 

0.6 

0.9994 

22.38 

1.24 

55.93 

54.17 

9.27 

0.47 

44.43 

3 

0.7 

0.9996 

22.56 

1.25 

56.00 

54.43 

8.03 

0.60 

45.80 

1 

0.6 

0.9994 

22.66 

1.27 

56.63 

54.80 

9.50 

0.70 

44.60 

2 

2 

0.6 

0.9996 

22.43 

1.24 

55.93 

54.17 

9.23 

0.43 

44.50 

3 

0.6 

0.9996 

22.62 

1.25 

56.00 

54.43 

8.23 

0.30 

45.90 

1 

0.7 

0.9994 

22.64 

1.27 

56.63 

54.80 

9.37 

44.87 

3 

2 

0.7 

0.9999 

22.39 

1.24 

55.93 

54.17 

8.93 

0.40 

44.83 

3 

0.7 

0.9998 

22.65 

1.26 

54.43 

8.03 

0.53 

45.87 

Table  333.  Fourth  Scenario  Results 
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Priority 

Exp.  Auctioneer  Increase 
Rate 

Fill 

Rate 

Average 
Time  an 
Ann. 

Spends  in 
Queue 

Average 
Number 
of  Ann.s 
in  Queue 

Total 
Number 
of  Ann.s 

Number 

of 

Resolved 

Ann.s 

First  Second  Third 

Round  Round  Round 

Resolved  Resolved  Resolved 

Sum  of  Sum  of  Sum  of 
Bidders  Bidders  Bidders 
1-3  4-6  7-9 

Share  Share  Share 

1 

0.1 

0.803 

33.13 

1.85 

56.63 

54.80 

3.43 

0.47 

50.77 

1  2 

0.1 

0.829 

32.85 

1.81 

55.93 

54.17 

3.93 

0.50 

49.70 

0.71 

0.22 

0.07 

3 

0.1 

0.796 

33.04 

1.82 

56.00 

54.43 

3.63 

0.57 

50.17 

1 

0.5 

0.970 

28.34 

1.58 

56.63 

54.90 

3.80 

2.70 

48.40 

2  2 

0.5 

0.971 

28.11 

1.55 

55.93 

54.47 

4.23 

2.77 

47.47 

0.38 

0.43 

0.18 

3 

0.5 

0.968 

28.43 

1.57 

56.00 

54.63 

3.83 

2.00 

48.80 

1 

0.1 

0.856 

33.13 

1.85 

56.63 

54.67 

3.47 

0.47 

50.73 

3  2 

0.1 

0.876 

32.86 

1.81 

55.93 

54.10 

3.67 

0.77 

49.67 

0.59 

0.30 

0.11 

3 

0.5 

0.875 

28.54 

1.58 

56.00 

54.67 

3.50 

2.30 

48.87 

1 

0.5 

0.923 

28.51 

1.59 

56.63 

54.90 

3.27 

3.07 

48.57 

4  2 

0.5 

0.924 

28.36 

1.56 

55.93 

54.47 

3.63 

2.67 

48.17 

0.49 

0.36 

0.15 

3 

0.1 

0.915 

33.09 

1.83 

56.00 

54.37 

3.43 

0.53 

50.40 

Table  344.  Comparison  of  Four  Scenarios 

Scenarios 

Fill  Rate 

Time  in  Q 

Number  In  Q 

First 

0.61 

11.80 

0.66 

Second 

0.11 

11.80 

0.66 

Third 

1.00 

22.56 

1.26 

Fourth 

0.81 

33.01 

1.83 

6.4  Results  and  Discussion 


Since  humanitarian  supply  chains  have  unique  characteristics  when  compared  to  corporate 
supply  chains,  the  environment  should  be  understood  before  assessing  the  contribution  of  a 
framework.  In  the  first  phase  of  experimental  study,  environmental  factors  were  conceived  and 
they  are  kept  as  constants  in  the  second  phase.  This  approach  enabled  an  examination  of  the 
auction  design  parameters  proposed  in  this  research  within  reasonable  environmental  conditions. 
Each  scenario  focused  on  a  different  parameter  to  represent  the  effect  of  that  particular 
parameter. 

The  framework  includes  some  design  parameters  which  can  easily  be  implemented  in 
disaster  relief  operations.  It  is  shown  in  the  experimental  study  that  the  priority  of  items  and 
weighted  priority  of  a  bundle  affect  the  fill  rate  of  an  announcement.  Bundling  of  items  hasn’t 
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been  studied  in  the  literature  as  much  as  the  single  indivisible  item  case  [9],  Ease  of  logistics  and 
announcement  options  can  be  used  in  software  like  HELIOS  to  reach  higher  efficiency  in 
resource  allocation.  Bidder  and  auctioneer  decision  making  introduced  competition  among 
bidders  and  among  auctioneers,  which  is  the  practical  case  in  procurement  operations. 

The  framework  is  not  intended  to  find  the  market-clearing  price  using  each  party’s  valuations 
of  each  item  type;  rather,  we  focus  on  the  item  type  and  quantity  allocation  from  the  sellers  to  the 
buyers.  The  exploratory  research  [10]  serves  the  purpose  of  conceiving  the  specifics  of  disaster 
relief  logistics.  The  framework  proposed  in  this  research  is  a  quantitative  and  holistic  model  that 
can  be  used  to  address  the  specific  needs  of  disaster  relief  logistics. 

The  network-flow  models  [2,11,19]  are  narrowly  focused  and  usually  deal  with  vehicle 
routing  and  allocation  of  specific  resources,  whereas  our  proposed  framework  is  holistic  (from 
demand  creation  to  the  fulfilment  of  the  demand)  as  well  as  modelling  the  procurement  activity 
with  an  auction.  Resource  allocation  problems  are  typically  solved  in  the  literature  [6,7,12]  for 
equipments,  vehicles,  and  reusable  supplies.  The  framework  here  provides  alternative  methods 
for  consumable  supplies;  therefore,  it  does  not  have  the  scheduling  components  that  are  found  in 
other  models. 

6.5  Conclusions 

A  simulation-based  procurement-auction  framework  is  presented  in  this  research  to  address 
the  inefficiencies  of  humanitarian  supply  chains.  In  humanitarian  supply  chains,  multiple  disaster 
locations  appeal  for  relief  items  at  the  same  time  in  an  area  where  supplier  resources  are  limited. 
The  specific  design  characteristics  of  disaster  relief  procurement  activities  are  incorporated  into 
the  Announcement  Construction,  Bid  Construction  and  Bid  Evaluation  phases  of  the  framework. 
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Disaster  locations  are  considered  as  auctioneers  and  suppliers  are  considered  as  bidders. 
Auctioneers  compete  to  one  another  in  multiple  rounds  of  the  procurement  auction.  The  holistic 
framework  in  three  phases  is  unique  not  only  in  procurement  auction  literature,  but  also  in 
disaster  relief  logistics.  The  value  notion  plays  a  balanced  role  in  the  framework,  since  the  bid 
construction  phase  minimizes  the  value,  but  the  bid  evaluation  phase  maximizes  the  value.  The 
use  of  the  value  notion  helps  suppliers  to  dispose  of  the  old  items  more  effectively  in  the  bid 
construction  phase  and  helps  the  disaster  location  to  get  better  conditioned  items  in  the  bid 
evaluation  phase. 

Disaster  locations  are  given  the  right  to  reject  the  bids  when  they  do  not  fulfil  a  certain 
portion  of  the  appeal  list.  When  disaster  locations  are  not  satisfied,  they  send  a  revised 
announcement  to  the  suppliers.  Disaster  locations  have  substitution  and  partial  fulfilment  options 
while  sending  the  announcement.  Together  with  quantity,  item  type  and  priority  information;  the 
announcement  options  give  a  complete  representation  of  the  appeals  list.  Multiple  rounds 
auctions  usually  require  bids  to  be  updated  in  each  round,  but  the  framework  in  this  research 
allows  disaster  locations  to  revise  the  announcements.  Priority  of  announcement  is  connected  to 
the  waiting  time  of  an  announcement.  Multiple  round  auctioning  helped  disaster  locations  to 
increase  their  fill  rate. 

Suppliers  are  better  evaluated  with  the  ease  of  logistics  parameter,  which  gives  importance  to 
the  suppliers  that  have  easy  access  to  the  disaster  location.  Suppliers  are  given  the  right  to  use 
bidding  strategies  when  the  announcement  is  not  urgent.  The  balance  with  urgency  in  disaster 
relief  operations  and  supplier  preferences  are  accomplished  with  some  threshold  levels.  Some 
bidding  strategies  performed  well  in  certain  settings,  which  leads  to  the  conclusion  that  these 
strategies  should  be  disaster  specific. 
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7.1  Executive  Summary 


We  consider  a  single-unit  system  operating  in  a  Markovian  environment  such  that  the 
instantaneous  rate  of  degradation  of  the  system  is  detennined  by  the  current  state  of  the 
environment.  The  system  fails  when  its  cumulative  degradation  passes  a  threshold.  We  utilized  a 
discrete-event  simulation  model  to  mimic  the  deterioration  and  failure  of  the  system  and  fit  a 
time  to  first  failure  distribution  to  the  simulated  data.  A  numerical  experiment  is  used  to  evaluate 
the  quality  of  the  fit  for  multiple  sets  of  system  parameters.  The  goodness  of  fit  test  results 
indicate  a  three-parameter  Weibull  distribution  as  a  reasonable  estimate  for  the  system.  We 
extend  this  concept  into  the  study  of  a  system  with  a  Weibull  random  variable  threshold.  The 
corresponding  fit  tests  indicate  a  Weibull  distribution  to  be  a  reasonable  estimate.  We  then 
develop  and  compare  an  age-replacement  policy  and  a  condition-based  maintenance  policy  based 
on  the  Weibull  approximation.  The  condition-based  plan  is  found  to  be  the  more  cost  efficient 
option. 

7.2  Introduction 

A  repairable  system  (RS)  is  a  unit  of  equipment  which  can  be  restored  to  an  operating 
condition  through  maintenance  actions  after  experiencing  a  failure.  Maintenance  actions  include 
those  intended  to  delay  system  failure  and  those  taken  in  response  to  failure.  These  maintenance 
actions  can  be  classified  into  three  categories:  routine  maintenance,  corrective  maintenance,  and 
preventive  maintenance.  Routine  maintenance  consists  of  the  daily  actions  performed  on  the 
system  to  ensure  proper  system  performance.  Corrective  maintenance  includes  actions  performed 
as  a  response  to  failure.  Preventive  maintenance  actions  are  intended  to  delay  system  failure. 
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Preventive  maintenance  can  be  further  divided  into  two  categories:  scheduled  maintenance  and 
predictive  maintenance. 

Using  a  perfect  predictive  maintenance  approach,  preventive  maintenance  would  be 
performed  only  when  absolutely  necessary  and  avoiding  all  failures.  Singpurwalla  [2]  presents 
stochastic  failure  time  models  based  on  failure-causing  mechanisms,  like  degradation  and  wear, 
which  are  fitting  for  a  repairable  system  in  a  dynamic  environment.  Kharoufeh  and  Cox  [1] 
present  a  model  in  which  the  system  is  operating  in  an  environment  represented  as  a  continuous¬ 
time  Markov  Chain  (CTMC).  The  rate  of  degradation  of  the  system  depends  on  the  current  state 
of  the  environment.  The  procedure  analyzes  the  reliability  and  estimates  the  lifetime  distribution 
of  a  RS  operating  in  such  an  environment. 

We  explore  the  potential  for  approximating  the  mathematically  complex  time  to  failure 
and  residual  life  distributions  resulting  from  the  Kharoufeh  and  Cox  [1]  approach  with  traditional 
time  to  failure  distributions  (e.g.  Weibull)  and  construct  age-based  and  condition-based  policies 
to  compare  the  economic  perfonnance  of  the  RS  under  each  policy.  We  extend  this  work  by 
assuming  the  deterioration  threshold  which  results  in  RS  failure  is  a  random  variable  with  a 
Weibull  distribution  instead  of  a  constant  value.  After  constructing  age  and  condition-based 
maintenance  policies  for  this  altered  system,  we  evaluate  and  compare  the  cost  per  unit  time  of 
the  repairable  system  under  each  policy. 
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7.3  Methods,  Assumptions,  and  Procedures 


7.4  System  definition 

Consider  a  repairable  system  belonging  to  the  type  modeled  by  Kharofeh  and  Cox  [1]. 
The  system  is  operating  in  a  stochastic  environment  which  is  modeled  as  a  three-state, 
continuous  time  Markov  chain.  X{t)  denotes  the  state  of  the  environment  at  time  t.  X(t)  is  bound 
by  the  following  two  constraints:  (1)  X(t)  6  {-1,0,1},  for  all  t>  0;  (2)  A(0)  =  0.  The  three  states 
correspond  to  the  following  operating  conditions  for  the  system:  state  0  is  considered  to  be 
“nominal”;  state  -1  is  considered  to  be  “less  severe  than  nominal”;  state  1  is  considered  to  be 
“more  severe  than  nominal.”  Leto  .denote  the  transition  rate  of  the  Markov  chain  from  state  x 

2  X,X 

to  state  x’  and  px  v,  represent  the  transition  probability  from  state  x  to  state  x’,  for  all  values  of 
xg  {-1,0,1}  and  x'e  {-1,0,1}  given  thatx^x'.  In  addition,  let  v  denote  the  transition  rate  out  of 
state  x  for  all  x  in  the  set  {- 1 ,  0,  1 } . 

The  system  experiences  stochastic  deterioration,  with  the  instantaneous  rate  of  system 
deterioration  dependent  on  the  current  state  of  the  environment.  Let  r(X(t))  describe  the 
instantaneous  rate  of  system  deterioration  at  time  t,  for  all  t>0-  To  better  facilitate  our 
definition  of  the  instantaneous  rate  of  system  deterioration,  we  assume  r(— 1)  <  r(0)  <  r(\) . 

Let  Y(t )  represent  the  cumulative  deterioration  of  the  system  at  time  t,t>  0.  Therefore, 

t 

Y(t)  =  J r(X ( u))du  • 
o 

The  system  fails  for  the  first  time  when  the  cumulative  deterioration  (7(0)  is  greater  than  or 
equal  to  the  failure  threshold  (y/).  Let  T  represent  the  time  to  first  failure,  therefore 
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T  =  mm{t  \Y(t)>y/}.  We  initially  assume^// is  a  constant  value  and  then  that  it  is  a  random 
variable  with  a  Weibull  distribution,  therefore  T  is  a  non-negative  random  variable. 

7.5  Analysis  of  time  to  first  failure  for  constant  threshold 

In  order  to  generate  the  values  of  T,  a  discrete-event  simulation  model  of  the  system  was 
built.  This  model  was  constructed  in  Visual  Basic,  and  begins  with  the  system  in  state  0.  The 
model  mimics  the  transitions  of  the  underlying  Markov  chain,  ends  when  the  cumulative 
deterioration  is  greater  than  or  equal  to  the  randomly  generated  failure  threshold,  and  then 
outputs  the  time  to  first  failure.  The  user  inputs  the  number  of  replications  that  the  model  runs, 
the  deterioration  rates  of  each  of  the  three  states  of  the  Markov  chain,  the  transition  rates  of  the 
Markov  chain,  and  the  deterioration  failure  threshold. 

Considering  a  specific  numerical  example  and  visual  inspection  of  the  histogram  formed 
from  1000  observations,  the  three-parameter  truncated  Weibull  distribution  was  found  to  be  a 
reasonable  candidate  for  approximating  the  probability  distribution  of  T.  Under  this  assumption, 
the  cumulative  distribution  function  of  T  is  initially  approximated  by 

^  ~  fmin 

l  V 

where 

t  .  =  JC_ 
min  ,■(+ 1) 

and  >  1  and  r/>  0.  Recognizing  that  T  <  tmax  ,  where 

t  =-*— 
max  r(-l) 

an  improved  approximation  of  the  cumulative  distribution  function  of  T  is  found  to  be 
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and  the  probability  density  function  of  T  can  be  better  approximated  by 


Assuming  the  probability  distribution  of  the  data  has  cdf  G(t),  the  values  of  ft  and  7  can 
be  estimated  using  the  likelihood  function 

n 

L(p,r])= 

Z=1 

and  the  log-likelihood  function 

A(/?,?;)=  ln(Z,(/?,;;)) . 

The  estimation  of  J3  and  77  can  then  be  obtained  by  simultaneously  solving  the  following  two 
equations: 

0 

dp 

8A{P^)  0_ 

dr/ 

This  optimization  problem  can  be  solved  using  the  cyclic  coordinate  method  of  numerical  search. 

The  final  step  in  the  analysis  of  the  data  is  to  utilize  a  Kolmogorov-Smirnoff 
(K-S)  [3]  test  to  compare  the  data  to  the  approximate  probability  distribution.  To  conduct  the 
test,  50  observations  of  T  are  sorted  into  ascending  order,  and  the  test  statistic,  JnD„ ,  is 
computed  using 
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=  max-  —  -  0(1(1])  >, 

1<!<«  n  u 

Z>,7  =  maxi  — -  >, 

l<i<«  '  ’  n 

and 

D„  =  ma  x\p-,D+). 

1  <i<n 

The  null  hypothesis,  that  T  is  a  random  variable  having  a  cumulative  distribution  function  G(t) 
with  parameters  f>  and  q  is  rejected  if 

■JnDn  >  C*_a 

where  C*_a  is  the  critical  value  for  a  level  of  significance  of  a  and  sample  size  n. 


7.6  Analysis  Of  Time  To  First  Failure  For  Random  Variable  Threshold 

We  used  many  of  the  same  methods  in  the  extension  from  a  constant  threshold  to  a 
random  variable  threshold.  We  found,  through  a  numerical  experiment  and  visual  inspection  of  a 
1 000  observation  histogram,  that  in  this  system  T  would  be  well  approximated  through  the  use  of 
a  Weibull  probability  distribution.  The  resulting  cumulative  distribution  function  is  given  by 

"  (tY 

F(t )  =  1  -  exp - 


for  p  >\  and  77  >  0 ,  and  the  probability  density  function  is 


Utilizing  the  same  method  described  previously,  we  find  that  estimates  of  ft  and//  can  be 
found  by  solving  the  following  two  equations  simulateously: 
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The  data  is  then  subjected  to  a  K-S  test  (as  described  previously)  in  order  to  compare  it  to 
the  approximate  probability  distribution,  the  Weibull. 


7.7  Age-Based  Maintenance  Policy 

The  approximate  probability  distribution  on  T  can  be  used  to  develop  an  age-based  PM 
policy  for  the  RS.  Instantaneous,  perfect  corrective  maintenance  is  perfonned  if  the  RS  fails.  If 
the  RS  operates  without  failure  for  r  time  units,  then  instantaneous,  perfect  preventive 
maintenance  is  perfonned.  The  cost  of  each  corrective  maintenance  action  is  cCM  and  the  cost  of 

each  preventive  maintenance  action  is  cPM  such  that  0  <  cPM  <  ccu .  Since  both  CM  and  PM 
restore  the  RS  to  good  as  new  condition,  the  maintenance  of  the  RS  can  be  modeled  as  a  renewal 
process  with  renewal  points  at  each  maintenance  occurrence.  Let  ju{t)  denote  the  expected 
duration  of  one  cycle  of  the  renewal  process  (duration  of  time  between  maintenance  actions), 
under  an  age-based  PM  policy  r  .  Then, 

/ «(r)=  [  tf(t)dt  +  r(l  -F(t))  (21) 

JO 

Let  p{ t)  denote  the  expected  value  of  the  cost  of  each  cycle  of  the  renewal  process  under  the 
same  age-based  policy.  Therefore, 

P(*)  =  ccmF{t)  +  cpm(\-F(t)).  (22) 

The  long-run  average  maintenance  cost  per  unit  time  (ySM  (r  ))  under  the  age-based  PM  policy  is 
given  by 
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(23) 


7sm  O’)  = 


P(j) 

M(t) 


In  order  to  compute  ySM  (r)  for  any  given  r ,  numerical  integration  is  required,  followed  by 
numerical  search  to  find  the  value  of  r  that  minimizes  the  long-run  cost. 


7.8  Condition-Based  Maintenance  Policy 

Assume  the  RS  is  one  in  which  inspection  can  be  used  at  any  time  to  determine  the 
current  value  of  the  cumulative  deterioration.  Instead  of  pursuing  an  age-based  PM  policy,  we 
also  consider  a  condition-based  PM  policy  based  on  periodic  inspection  of  the  system. 
Instantaneous,  perfect  CM  is  perfonned  if  the  system  fails,  and  the  cumulative  deterioration  Y(t) 
is  observed  at  t  =  8, 28, ...{8  >  0) .  If  the  cumulative  deterioration  is  greater  than  a  certain  threshold 
(£  )  at  any  time  of  inspection,  then  the  system  undergoes  instantaneous,  perfect  PM.  The  cost  of 
each  inspection  is  cI ,  and  is  related  to  the  costs  of  PM  and  CM  by  0  <  c7  <  cPM  <  ccu .  In  order  to 
compare  the  two  policies,  we  calculate  yCBM  (8,  g) ,  the  long-run  average  cost  per  unit  time  under 
the  condition-based  maintenance  policy.  In  order  to  evaluate  the  average  cost  per-unit  time,  we 
developed  a  discrete  event  simulation  model. 

7.9  Results  and  Discussion 


7.10  Constant  Failure  Threshold 

Consider  an  example  system  such  that  r(-l)  =  0.8,  r( 0)  =  1.05,  r(+l)  =  1.25,  <7-1,0  =  40, 
<70,-1  =  35,  <7o,+i  =  80,  <7+1,0  =  45,  and  y/=  1.  The  simulation  model  was  used  to  generate  1000 
observations  on  the  time  to  first  failure  of  the  system.  Following  the  methods  described  in  the 
previous  section,  the  resulting  estimates  of  [1  and  i]  are  given  by  4.0067  and  0.1421,  respectively. 


141 


For  this  example,  we  utilized  50  observations  in  order  to  perform  the  K-S  test  and  found  that 
y[50Dso  =  0.811 ,  which  is  less  than  Co  95  =0.856  when  the  sample  size  is  50  and  a  =  0.05.  This  result 
indicates  a  failure  to  reject  the  null  hypothesis,  indicating  that  a  claim  could  be  made  that  the 
truncated  three  parameter  Weibull  distribution  provides  a  reasonable  fit  to  the  data. 

In  order  to  generalize  these  results  to  a  wider  range  of  system  parameters,  we  developed  a 
larger  numerical  experiment.  Assuming  the  failure  threshold  y/=  1 ,  the  system  degradation  rates 
are  drawn  from  uniform  probability  distributions  as  shown: 
r(0)~U(0.8,1.2) 
r(-l)  ~  U(0.25,0.75)  x  r(0) 
r(+l)~U(1.5,4)  xr(0), 

and  the  transition  rates  are  developed  according  to 
Oo  ~  U(100,150) 

V-\  ~  U(1.5,4)  xo0 
o+i  ~  U(1.5,4)  x  o0. 

Similarly,  the  transition  probabilities  are  found  according  to 
P- i,o  =  1 
P+ 1,0  =  1 

P0,-1  ~  U(0.25,0.75) 

Po,+i  =  1  ~Po,-u 

and  the  transition  rate  can  be  obtained  from 

<7x,x'  =  Px,x'vx 

for  allx  =  -1,  0,  +1,  x'  =  -1,  0,  +1,  x'  *x. 
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This  system  was  used  to  develop  10,000  experimental  systems,  each  with  50 
observations.  For  each  data  set,  the  parameters  of  the  Weibull  distribution  were  estimated  and  a 
K-S  test  was  performed.  The  null  hypothesis  was  rejected  only  3289  times  in  10,000  tests  with 

a  =  0.05  . 

Consider  an  example  system  such  that  r{- 1)  =  0.6,  r(0)  =  1,  r(+l)  =  3.5,  <7-1,0  =  40,  <70,-1  = 
35,  </(),+]  =  80,  <7+1,0  =  45,  and  y/=  1.  Suppose  ccm  =  1,  cpm  =  0.8,  and  c/  =  0.02.  The  estimated 
Weibull  parameters  for  this  system  are/?=  2.8,  and  77  =  0.23. 

This  system  yields  a  recommended  age-based  preventive  maintenance  policy  of  x  =  1.5 
and  a  recommended  condition-based  policy  of  8=  0.44  and  <^=  0.46.  Using  50  replications,  each 
with  a  run  length  of  100  time  units,  we  estimate  /sm(  1-5)  and  yc«.v/(0.44,0.46).  According  to  these 
policies,  the  estimated  average  maintenance  cost  under  the  age-based  policy  is  2.85,  and  the 
estimated  average  maintenance  cost  under  the  condition-based  policy  is  2.10.  This  indicates  an 
estimated  savings  of  26%  through  the  usage  of  a  condition-based  approach  over  a  scheduled 
approach. 

7.11  Random  Failure  Threshold 

Consider  the  following  example:  r(— 1)  =  0.6 ,  r(0)  =  l,  r(+l)  =  3.5,  <7_10=40,  q0_{=35, 
q0+l  =80,  and  q+l0  =45.  Suppose  the  failure  threshold  is  a  Weibull  random  variable  having  a 

shape  parameter  of  2.5  and  a  mean  of  1.  The  simulated  model  was  executed  with  these  inputs 
and  1000  observations  on  T  were  generated.  The  values  of  the  Weibull  parameters,  /3  and  //  , 
were  2.291  and  0.607  respectively.  For  this  example,  V50£>50  =  0.811.  Choosing  a  level  of 
significance  of  0.05,  C0*95  =  0.856  .  Since  0.81 1<0. 856,  the  null  hypothesis  fails  to  be  rejected  and 
the  Weibull  distribution  appears  to  be  a  good  estimate  of  the  time  to  first  failure. 
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To  generalize  these  results  to  a  wider  range  of  parameters,  we  developed  a  numerical 

experiment.  We  generated  10,000  data  sets  of  50  observations  each  with  failure  threshold  i// 

distributed  as  a  Weibull  random  variable  with  mean  time  to  failure  of  one.  The  system 

degradation  rates,  transition  values,  and  transition  probabilities  are  shown  below: 

r(-l)  ~  U (0.25,0.75)  *  r(0) 
r(0)  ~  U (0.75,1.25) 
r(+l)  ~  U (1.5,4)  *  r(0) 

v( — 1)  —  C/(l  .5,4)  *  v(0) 
v(0)  ~  £7(100,200) 
v(+l)  ~  (7(1 .5,4)  *  v(0) 

P- 1.0  = 1 
P+ i,o  = 1 

p0  _x  ~  £7(0.25,0.75) 

Pom  = 1  -  Pom 

and  the  transition  rates  for  the  system  can  be  obtained  by 

4*,*'  =  TV  *  Av*  e  [-  h0,+l]x  *  x' . 

For  each  of  the  50  data  sets,  Weibull  distribution  parameters  were  found  and  a  K-S  test  was 
performed.  At  a  =  ().()5 ,  the  null  hypothesis  was  rejected  only  1446  times,  indicating  a  pass 
percentage  of  85.54%  and  confirming  the  Weibull  model  as  a  good  estimation  of  the  time  to  first 
failure. 

Continuing  the  example  in  which  r(— 1)  =  0.6 ,  r(0)  =  l,  r(+l)  =  3.5,  g_10=40,  q0_t  -  35, 
q0  +1  =  80 ,  q+x  o  =  45 ,  and  the  failure  threshold  modeled  as  a  Weibull  distribution  with  a  mean 

time  to  failure  of  1  and  parameters,  ,  /T  and  >],  equal  to  2.291  and  0.607  respectively,  we  develop 
maintenance  policies  with  cCM  =  1  ,cPM  =  0.8  and  c,  =  0.02 . 
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This  system  yields  a  recommended  age-based  preventive  maintenance  policy  of  r  =0.814 
and  a  recommended  condition-based  preventive  maintenance  policy  of  S  =0.666  £=0.579.  A 
discrete  event  simulation  model  was  used  to  run  50  replications  of  100  time  units.  The  age-based 
policy  indicated  an  average  cost  per  time  unit  of  4.64,  and  the  condition-based  policy  yielded  a 
cost  per  time  unit  of  4.08.  These  results  indicate  a  possible  savings  of  13.73%  by  choosing  a 
condition-based  policy  of  an  age-based  policy. 

7.12  Conclusions 

In  this  study,  we  present  an  approach  for  the  estimation  of  full  lifetime  distributions  for 
single-unit  systems  operating  in  a  Markovian  environment.  Our  results  indicate  the  this  approach 
successfully  provides  a  reasonable  approximation  of  the  time  to  first  failure  distribution  when 
considering  both  constant  value  failure  thresholds  and  random  variable  failure  thresholds. 

In  addition,  we  develop  age  and  condition-based  preventive  maintenance  policies  for 
single-unit  repairable  systems  for  both  constant  and  random  failure  thresholds.  Our 
experimentation  provides  evidence  that  there  is  cost  reduction  possible  by  shifting  from  a 
scheduled  maintenance  program  to  one  based  on  condition-based  maintenance. 

7.13  Recommendations 

Possible  extensions  to  this  research  include  the  consideration  of  multi-unit  systems  to 
further  generalize  the  results  found.  In  addition,  a  model  utilizing  data  that  is  only  correlated  to 
deterioration  would  provide  a  more  realistic  look  into  how  well  this  approach  to  modeling 
overall  system  degradation  would  work  in  real-world  situations. 
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8.1  Executive  Summary 

Statistical  process  control  has  been  considered  along  with  generalized  linear  models  of 
influenza  activity.  Different  combinations  of  distribution  and  link  assumptions  have  been 
formally  compared  in  order  to  choose  one  that  would  seem  theoretically  appropriate.  Specificity 
of  gamma  regression-based  individuals  control  charts  for  influenza  activity  has  been  estimated 
and  compared  to  begin  answering  the  important  question:  What  are  reasonable  expectations 
about  errors  associated  with  outbreak  detection? 

Results  include  performance  among  gamma  regression-based  charts  for  laboratory  data 
that  is  relatively  near  what  one  should  expect  from  traditional  process  control.  It  is  shown  how 
more  theoretically  appropriate  statistical  quality  control  models  of  flu  activity  become 
increasingly  specific  and  so  useful  in  identifying  non-seasonal  outbreaks.  And  it  is  shared  how 
this  research  is  being  applied  in  practice. 
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8.2  Introduction 


High-profile  disease  outbreaks  and  the  threat  of  terrorism  have  increased  interest  in  public  health 
surveillance  systems  for  early  outbreak  detection  (Buehler  et  ah,  2004).  Surveillance  systems  as 
defined  by  Centers  for  Disease  Control  and  Prevention  (CDC)  collect  and  analyze  morbidity  and 
other  data  and  facilitate  timely  dissemination  of  results  to  the  appropriate  decision-makers 
(Bravata  et  al,  2004).  However,  the  issue  of  surveillance  will  be  considered  here  as  one  of 
analysis  and  interpretation  of  health  data.  Also  one  may  distinguish  between  what  public  health 
surveillance  is  syndromic  and  what  is  better  known  as  spatial  (Kleinman  et  al.,  2004;  Lawson 
and  Kleinman,  2005).  Syndromes  are  groups  of  symptoms  that  combine  similar  types  of 
complaints,  and  surveillance  generally  refers  to  monitoring  statistics  for  evidence  of  change. 
Spatial  techniques  are  used  to  find  clustering  across  a  map. 

Data  used  for  illustration  here  will  be  that  of  influenza  (flu)  activity  collected  and 
reported  by  the  Influenza  Branch  at  CDC.  Data  of  influenza-like  illnesses  (ILI)  are  of  particular 
interest,  because  they  include  a  variety  of  illnesses  with  similar  clinical  symptoms,  including 
anthrax  (Valleron  and  Vidal,  2002).  For  example,  an  anthrax  outbreak  might  be  detectable  at  first 
as  an  unusual  increase  in  patients  reporting  emergency  departments  with  respiratory  symptoms 
(Reis  and  Mandl,  2003).  Through  acknowledging  the  importance  of  broader  research  into 
surveillance  networks  (Morse  et  al.,  2003),  programs  (Yih  et  al.,  2004),  and  systems 
(Widdowson  et  al.,  2003;  Paladini,  2004;  Zhang  et  al.,  2004;  Dafni  et  al.,  2004;  Williamson  and 
Hudson,  1999),  the  objective  of  the  work  described  here  has  been  to  measure  the  specificity  of 
combining  statistical  process  control  (SPC)  and  generalized  linear  models  (GLM)  for  outbreak 
detection. 


149 


8.3  Methods,  Assumptions,  and  Procedures 

Let  us  assume  that  the  objective  for  specificity  should  be  that  the  probability  of  finding  an 
outbreak  in  error  is  0.0027,  which  is  associated  with  traditional  SPC.  What  we  will  find  is  that 
based  on  algorithms  presented  here,  the  probability  0.0027  would  be  a  remarkable  underestimate. 
And  so  we  are  left  with  what  might  be  the  most  important  question  of  this  study:  What  are 
reasonable  expectations  about  errors  associated  with  outbreak  detection? 

In  addition  to  results  of  an  SPC-only  approach  there  will  be  presented  results  about  SPC 
applied  to  absolute  values  of  residuals  of  both  linear  and  gamma  regression.  Linear  regression 
assumes  a  nonnal  response,  where  its  mean  is  a  linear  function  of  predictors.  For  gamma 
regression,  we  have  assumed  a  gamma-distributed  response  where  the  natural  logarithm  of  the 
mean  is  a  linear  function  of  predictors,  which  is  also  known  as  the  log-gamma  model  (Hardin 
and  Hilbe,  2001).  Where  yt  represents  each  observation  of  the  response  variable,  x,f)  is  the  linear 
combination  of  covariate  values  and  associated  parameters,  and  (/)  is  the  scale  required  to  produce 
standard  errors  following  a  gamma  distribution,  the  log-likelihood  function  for  the  log-gamma 
model  is 
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Stata  statistical  software  has  been  used  for  this  maximum  likelihood  estimation  (StataCorp, 
2005). 

Regression  models  of  lab  data  take  as  the  response  total  isolates;  predictors  are  number  of 
labs  reporting,  total  specimens  tested,  and  dummy  variables  describing  categories  for  virus  and 
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region.  Models  of  sentinel  data  take  total  ILI  as  a  function  of  total  patients,  number  of  providers 
reporting  and  variables  describing  age  and  region.  Again  the  flu  activity  data  are  collected  in 
time  (weekly). 

In  addition  to  the  simple  individuals  control  chart  only  approach,  linear  regression  was 
chosen  for  its  familiarity.  A  GLM  philosophy  led  to  choosing  gamma  regression.  Each  of  five 
datasets  was  used  to  fit  appropriate  combinations  of  models  assuming  different  response 
distributions  and  functions  for  their  means.  The  distributions  are  gamma,  negative  binomial, 
nonnal,  and  Poisson;  the  functions  are  identity,  inverse,  and  natural  logarithm.  Maximum 
likelihood  among  the  combinations  was  found  in  every  case  to  be  what  we  have  referred  to  as  the 
gamma  regression  with  log  link. 

8.4  Results  and  Discussion 

"The  likelihood  function  is  defined  as  the  model  taken  as  a  function  of  the  unknown  parameter 
vector  for  the  given  observed  value  of  y.  A  model  that  makes  the  observed  data  more  probable  is 
said  to  be  more  likely  (Lindsey,  1997,  p.  221)."  An  additionally  chosen  diagnostic  here  relies  on 
the  concept  of  Type  I  error  that  places  more  traditional  goodness  of  fit  squarely  in  the  context  of 
SPC.  Further,  "the  study  of  residuals  [for  model  checking]  is  generally  only  useful  if  the  sample 
size  is  relatively  small,  at  most  100  observations  or  so  (Lindsey,  1997,  p.  221)."  Sample  sizes 
here  are  between  1000  and  5000.  Formal  comparisons  that  led  to  the  choice  of  log-gamma  over 
other  GLM  were  according  to  maximum  log-likelihood.  The  relevant  values  appear  in  Table  35, 
and  the  reader  can  verily  that  the  log-gamma  model  is  consistently  best  fitting  and  so  assumedly 
the  most  theoretically  appropriate. 
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Table  35.  Log  likelihood  Values  Associated  with  Main  Effects  GLM  given  Reasonable 

Combinations  of  Distribution  and  Link 


Distribution 

Link 

Lab 

97-01 

Lab 

01-03 

Sentinel 

97-01 

Sentinel 

02-03 

Sentinel 

03-05 

Gamma 

Log 

-971 

-4785 

-9060 

-8198 

-14,623 

Gamma 

Identity 

-1.71E10 

-3.30E9 

-7.12E10 

-3.87E10 

-2.64E11 

Neg.  bin. 

Identity 

-18,011 

-9852 

-21,908 

-15,500 

-63,934 

Neg.  bin. 

Reciprocal 

-125,752 

-71,112 

-9217 

-18,195 

-50,008 

Gaussian 

Reciprocal 

-23,597 

-14,509 

-11,002 

-11,681 

-18,271 

Poisson 

Identity 

-68,277 

-33,868 

-108,741 

-52,663 

-236,853 

Poisson 

Reciprocal 

-139,422 

-97,876 

-752,214 

-266,397 

-750,771 

Moving  ranges  are  correlated,  but  individual  measurements  are  assumedly  uncorrelated 
which  contradicts  what  is  known  about  time  series.  On  the  other  hand,  the  individuals  control 
chart  is  insensitive  to  small  shifts  that  would  seem  advantageous  to  outbreak  detection  and 
requisite  seasonality.  Were  it  not  interesting  to  compare  process  control  of  flu  activity  across 
model-based  statistics,  one  might  have  chosen  more  theoretically  appropriate  methods  such  as  P 
charts  for  proportions  or  ratios,  as  they  have  been  called  here. 

Lab  and  sentinel  data  have  been  further  separated  for  analysis  according  to  how  they  have 
been  collected  over  the  years.  Lab  data  collection  changed  somewhat  during  2001,  so  lab 
datasets  analyzed  here  are  1997-2001  and  2001-2003.  Sentinel  data  collection  changed  at  the  end 
of  2001  and  again  in  2003,  so  sentinel  results  are  for  1997-2001,  2002-2003,  and  2003-2005. 
More  recent  lab  data  was  not  available  at  the  time  of  this  first  analysis. 

Table  36  gives  example  details  of  the  log-gamma  model  for  2001-2003  lab  data  (log 
likelihood  is  -4785.334406).  Baseline  has  virus  is  A  (HI)  and  region  is  New  England. 
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Table  35.  Example  Details  of  the  Log-Gamma  Model  for  2001-2003  Lab  Data 


Total  isolates 

Coefficient 

Standard  error 

Z 

Specimens  tested 

0.0055275 

0.0005708 

9.68 

Labs  reporting 

0.3397659 

0.0350394 

9.70 

Virus  is  A  (H3N2) 

1.031436 

0.182429 

5.65 

Virus  is  A  (unknown) 

1.229036 

0.1750723 

7.02 

Virus  is  B 

1.332141 

0.1725199 

7.72 

Region  is  Mid-Atlantic 

-2.708092 

0.2797416 

-9.68 

Region  is  EN  Central 

-3.139738 

0.3134402 

-10.02 

Region  is  WN  Central 

-2.943873 

0.2942121 

-10.01 

Region  is  S  Atlantic 

-3.22848 

0.2867831 

-11.26 

Region  is  ES  Central 

-1.226358 

0.259689 

-4.72 

Region  is  WS  Central 

-2.024777 

0.2757869 

-7.34 

Region  is  Mountain 

-2.911517 

0.3184262 

-9.14 

Region  is  Pacific 

-2.939222 

0.3056996 

-9.61 

Constant 

-3.514507 

0.3074684 

-11.43 

Lind  in  Table  37  the  following  columns  described  left  to  right. 

1)  Lab  and  sentinel  datasets  and  years  included  (column  is  “Dataset”) 

2)  Regional,  weekly  observations  of  different  viruses  and  ages  (column  is  “Number  of 
Observations”) 

3)  Appropriate  ratios  and  associated  moving  ranges  simultaneously  outside  three  sigma 
limits  of  individuals  and  moving  ranges  control  charts  (column  is  “SPC  alone”) 
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4)  Linear  regression  residuals  and  associated  residuals  simultaneously  outside  three  sigma 
limits  of  individuals  and  moving  ranges  control  charts  (column  is  “Linear  regress  +”) 

5)  Gamma  regression  residuals  and  associated  residuals  simultaneously  outside  three  sigma 
limits  of  individuals  and  moving  ranges  control  charts  (column  is  “Gamma  regress  +”) 

The  proportions  of  Table  37  in  columns  3,  4,  and  5  should  be  thought  of  as  probabilities  of  Type 
I  error  or  finding  an  outbreak  in  error. 

Table  36. Probabilities  of  Finding  an  Outbreak  in  Error  for  SPC  Alone,  Linear  Regress  +,  and 


Gamma  Regress  + 


Dataset 

Number  of 

observations 

SPC  alone 

Linear  regress  + 

Gamma  regress  + 

Lab  97-01 

4784 

0.0648 

0.0364 

0.0077 

Lab  01-03 

3060 

0.0575 

0.0386 

0.0069 

Sentinel  97-01 

1494 

0.0368 

0.0375 

0.0542 

Sentinel  92-03 

1872 

0.0438 

0.0342 

0.0577 

Sentinel  03-05 

2556 

0.0556 

0.0376 

0.0278 

Among  the  lab  data  with  respect  to  Type  I  error,  there  are  consistent  results:  gamma 

regression  with  SPC  outperforms,  and  SPC  alone  underperforms.  There  is  no  consistent  result 
among  sentinel  data.  Further,  the  outperforming  gamma  regression-based  algorithm  for  lab  data 
is  the  only  one  that  is  relatively  close  to  the  otherwise  expected  Type  I  error  probability  0.0027. 
Type  I  errors  associated  with  sentinel  data  are  generally  great  perhaps  because  they  are  more 
subjective  in  nature.  There  would  seem  to  be  less  measurement  (human)  error  associated  with  lab 
data  collection. 

In  the  case  of  lab  data  one  might  imagine  successively  more  theoretically  appropriate 
models  of  flu  activity  "explaining  away"  more  and  more  variation  before  moving  to  statistical 
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control  charts.  Imagine  the  models  "pushing"  more  observations  to  be  inside  upper  control  limits 
or  "pulling"  those  limits  outside  of  more  observations.  In  other  words,  we  are  moving  to  better 
fitting  models  of  flu  activity.  First,  with  SPC  alone  we  are  controlling  for  denominators  of  the 
ratios  charted,  number  of  labs  reporting,  or  total  patients.  As  we  move  to  regression-based  SPC, 
we  account  for  whatever  else  is  available  from  the  CDC,  and  within  this  framework:  The  gamma 
distribution  might  seem  more  appropriate  than  normal  for  absolute  values  of  any  kind;  the 
natural  logarithm  as  a  function  is  presumably  better  here  than  the  identity. 

8.5  Conclusions 

To  finish  on  a  note  regarding  strict  application,  I  want  to  quote  the  U.S.  Air  Force  Captain  who 

oversaw  the  technical  details  of  the  research  project  described  in  this  report: 

Syndromic  Surveillance  Systems  led  by  Dr  Justin  Chimka  will  offer  benefits  to  the 
medical  emergency  response  community  and  to  the  DOD  by  offering  a  framework  to 
evaluate  the  effectiveness  of  surveillance  systems  in  detecting  the  outbreak  of  specified 
illnesses.  The  procedures  developed  for  testing  U.S.  surveillance  data  will  be  applied  by 
AFRL/HEAL  to  influenza  like  data  collected  by  Air  Force  Institute  for  Operational 
Health  to  aid  the  DOD  Worldwide  Influenza  Surveillance  Program  study  of  identifying 
data  that  are  unusable  for  outbreak  reporting.  (Air  Force  Research  Laboratory,  Jean- 
Claude  Beasley,  March  15,  2006) 

8.6  Recommendations 

In  preliminary  research  for  a  follow  up  study,  no  different  lab  results  were  found  among 
more  recent  datasets  2003-2004  and  2004-2005.  Also,  simple  autoregressive  models  of  lab  data 
have  not  been  found  to  interfere  with  aforementioned  notions  of  what  is  good  for  lab-based 
surveillance.  Sentinel  data,  on  the  other  hand,  have  found  in  autoregressive  models  those  that 
consistently  outperfonn.  So  let  us  begin  discussion  of  future  work  with  hope  for  sentinel  data  of 
self-exciting  autoregressive  models  or  threshold  autoregressive  processes  that  do  even  more  to 
account  for  serial  correlation  and  seasonality.  The  addition  of  mathematical  models  of 
information  processing  such  as  artificial  neural  networks  has  also  been  considered. 


155 


Future  work  should  include  measurement  of  sensitivity  and  research  into  integrated 
models  for  broader  bio-terror  response  logistics  (Craft  et  ah,  2005)  and  mitigation  strategies  for 
pandemic  influenza  (Gennann,  et  ah,  2006).  Improved  outbreak  detection  for  bio-terror  response 
logistics  would  positively  affect  medical  materiel  transportation  forecasts,  cognitive  decision 
support  for  syndromic  surveillance  systems,  and  epidemic  models  for  anticipatory  and 
responsive  support;  that  is,  sense  and  respond  operations,  intelligence,  and  logistics. 

Future  data  to  be  more  systematically  considered  might  include  vaccine  usage  as  a 
predictor,  to  further  explain  the  variation  in  flu  activity  before  representing  what  is  left  with 
control  charts.  There  is  interest  in  better  understanding  variation  associated  with  OTC 
medication  sales  for  public  health  surveillance  (Goldenberg  et  ah,  2002;  Wagner  et  ah,  2004). 

However,  what  is  most  important  at  this  time  is  thought  to  be  the  problem  of  estimating 
Type  II  error  or  sensitivity  associated  with  our  detection  algorithms.  Related  problems  facing  the 
analyst  would  be  the  seemingly  important  gamma  distribution  and  its  computational  intensity 
compared  to  the  nonn.  Also,  not  only  is  there  serial  correlation  in  weekly  series  but  covariance 
of  individuals  and  moving  ranges.  Finally,  future  work  with  lab  data  might  be  most  important  to 
understanding  variation  of  influenza  A  viruses,  of  which  avian  influenza  or  "bird  flu"  is  a 
subtype.  While  both  A  and  B  viruses  concern  human  health,  only  type  A  can  cause  pandemics. 
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